WebGL Development Skill

File Organization: This skill uses split structure. See references/ for advanced patterns and security examples.

1. Overview

This skill provides WebGL expertise for creating custom shaders and visual effects in the JARVIS AI Assistant HUD. It focuses on GPU-accelerated rendering with security considerations.

Risk Level: MEDIUM - Direct GPU access, potential for resource exhaustion, driver vulnerabilities

Primary Use Cases:

• Custom shaders for holographic effects
• Post-processing effects (bloom, glitch)
• Particle systems with compute shaders
• Real-time data visualization

2. Core Responsibilities

2.1 Fundamental Principles

1. TDD First: Write tests before implementation - test shaders, contexts, and resources
2. Performance Aware: Optimize GPU usage - batch draws, reuse buffers, compress textures
3. GPU Safety: Implement timeout mechanisms and resource limits
4. Shader Validation: Validate all shader inputs before compilation
5. Context Management: Handle context loss gracefully
6. Performance Budgets: Set strict limits on draw calls and triangles
7. Fallback Strategy: Provide non-WebGL fallbacks
8. Memory Management: Track and limit texture/buffer usage

3. Technology Stack & Versions

3.1 Browser Support

3.2 Security Considerations

typescript
1// Check WebGL support and capabilities
2function getWebGLContext(
3  canvas: HTMLCanvasElement,
4): WebGL2RenderingContext | null {
5  const gl = canvas.getContext("webgl2", {
6    alpha: true,
7    antialias: true,
8    powerPreference: "high-performance",
9    failIfMajorPerformanceCaveat: true, // Fail if software rendering
10  });
11
12  if (!gl) {
13    console.warn("WebGL 2.0 not supported");
14    return null;
15  }
16
17  return gl;
18}

4. Implementation Patterns

4.1 Safe Shader Compilation

typescript
1// utils/shaderUtils.ts
2
3// ✅ Safe shader compilation with error handling
4export function compileShader(
5  gl: WebGL2RenderingContext,
6  source: string,
7  type: number,
8): WebGLShader | null {
9  const shader = gl.createShader(type);
10  if (!shader) return null;
11
12  gl.shaderSource(shader, source);
13  gl.compileShader(shader);
14
15  if (!gl.getShaderParameter(shader, gl.COMPILE_STATUS)) {
16    const error = gl.getShaderInfoLog(shader);
17    console.error("Shader compilation error:", error);
18    gl.deleteShader(shader);
19    return null;
20  }
21
22  return shader;
23}
24
25// ✅ Safe program linking
26export function createProgram(
27  gl: WebGL2RenderingContext,
28  vertexShader: WebGLShader,
29  fragmentShader: WebGLShader,
30): WebGLProgram | null {
31  const program = gl.createProgram();
32  if (!program) return null;
33
34  gl.attachShader(program, vertexShader);
35  gl.attachShader(program, fragmentShader);
36  gl.linkProgram(program);
37
38  if (!gl.getProgramParameter(program, gl.LINK_STATUS)) {
39    const error = gl.getProgramInfoLog(program);
40    console.error("Program linking error:", error);
41    gl.deleteProgram(program);
42    return null;
43  }
44
45  return program;
46}

4.2 Context Loss Handling

typescript
1// composables/useWebGL.ts
2export function useWebGL(canvas: Ref<HTMLCanvasElement | null>) {
3  const gl = ref<WebGL2RenderingContext | null>(null);
4  const contextLost = ref(false);
5
6  onMounted(() => {
7    if (!canvas.value) return;
8
9    // ✅ Handle context loss
10    canvas.value.addEventListener("webglcontextlost", (e) => {
11      e.preventDefault();
12      contextLost.value = true;
13      console.warn("WebGL context lost");
14    });
15
16    canvas.value.addEventListener("webglcontextrestored", () => {
17      contextLost.value = false;
18      initializeGL();
19      console.info("WebGL context restored");
20    });
21
22    initializeGL();
23  });
24
25  function initializeGL() {
26    gl.value = getWebGLContext(canvas.value!);
27    // Reinitialize all resources
28  }
29
30  return { gl, contextLost };
31}

4.3 Holographic Shader

glsl
1// shaders/holographic.frag
2#version 300 es
3precision highp float;
4
5uniform float uTime;
6uniform vec3 uColor;
7uniform float uScanlineIntensity;
8
9in vec2 vUv;
10out vec4 fragColor;
11
12void main() {
13  // Scanline effect
14  float scanline = sin(vUv.y * 200.0 + uTime * 2.0) * 0.5 + 0.5;
15  scanline = mix(1.0, scanline, uScanlineIntensity);
16
17  // Edge glow
18  float edge = smoothstep(0.0, 0.1, vUv.x) *
19               smoothstep(1.0, 0.9, vUv.x) *
20               smoothstep(0.0, 0.1, vUv.y) *
21               smoothstep(1.0, 0.9, vUv.y);
22
23  vec3 color = uColor * scanline * edge;
24  float alpha = edge * 0.8;
25
26  fragColor = vec4(color, alpha);
27}

4.4 Resource Management

typescript
1// utils/resourceManager.ts
2export class WebGLResourceManager {
3  private textures: Set<WebGLTexture> = new Set();
4  private buffers: Set<WebGLBuffer> = new Set();
5  private programs: Set<WebGLProgram> = new Set();
6
7  private textureMemory = 0;
8  private readonly MAX_TEXTURE_MEMORY = 256 * 1024 * 1024; // 256MB
9
10  constructor(private gl: WebGL2RenderingContext) {}
11
12  createTexture(width: number, height: number): WebGLTexture | null {
13    const size = width * height * 4; // RGBA
14
15    // ✅ Enforce memory limits
16    if (this.textureMemory + size > this.MAX_TEXTURE_MEMORY) {
17      console.error("Texture memory limit exceeded");
18      return null;
19    }
20
21    const texture = this.gl.createTexture();
22    if (texture) {
23      this.textures.add(texture);
24      this.textureMemory += size;
25    }
26    return texture;
27  }
28
29  dispose(): void {
30    this.textures.forEach((t) => this.gl.deleteTexture(t));
31    this.buffers.forEach((b) => this.gl.deleteBuffer(b));
32    this.programs.forEach((p) => this.gl.deleteProgram(p));
33    this.textureMemory = 0;
34  }
35}

4.5 Uniform Validation

typescript
1// ✅ Type-safe uniform setting
2export function setUniforms(
3  gl: WebGL2RenderingContext,
4  program: WebGLProgram,
5  uniforms: Record<string, number | number[] | Float32Array>,
6): void {
7  for (const [name, value] of Object.entries(uniforms)) {
8    const location = gl.getUniformLocation(program, name);
9    if (!location) {
10      console.warn(`Uniform '${name}' not found`);
11      continue;
12    }
13
14    if (typeof value === "number") {
15      gl.uniform1f(location, value);
16    } else if (Array.isArray(value)) {
17      switch (value.length) {
18        case 2:
19          gl.uniform2fv(location, value);
20          break;
21        case 3:
22          gl.uniform3fv(location, value);
23          break;
24        case 4:
25          gl.uniform4fv(location, value);
26          break;
27        case 16:
28          gl.uniformMatrix4fv(location, false, value);
29          break;
30      }
31    }
32  }
33}

5. Implementation Workflow (TDD)

5.1 Step-by-Step Process

1. Write failing test -> 2. Implement minimum -> 3. Refactor -> 4. Verify

typescript
1// Step 1: tests/webgl/shaderCompilation.test.ts
2import { describe, it, expect, beforeEach } from "vitest";
3import { compileShader } from "@/utils/shaderUtils";
4
5describe("WebGL Shader Compilation", () => {
6  let gl: WebGL2RenderingContext;
7
8  beforeEach(() => {
9    gl = document.createElement("canvas").getContext("webgl2")!;
10  });
11
12  it("should compile valid shader", () => {
13    const source = `#version 300 es
14      in vec4 aPosition;
15      void main() { gl_Position = aPosition; }`;
16    expect(compileShader(gl, source, gl.VERTEX_SHADER)).not.toBeNull();
17  });
18
19  it("should return null for invalid shader", () => {
20    expect(compileShader(gl, "invalid", gl.FRAGMENT_SHADER)).toBeNull();
21  });
22});
23
24// Step 2-3: Implement and refactor (see section 4.1)
25// Step 4: npm test && npm run typecheck && npm run build

5.2 Testing Context and Resources

typescript
1describe("WebGL Context", () => {
2  it("should handle context loss", async () => {
3    const { gl, contextLost } = useWebGL(ref(canvas));
4    gl.value?.getExtension("WEBGL_lose_context")?.loseContext();
5    await nextTick();
6    expect(contextLost.value).toBe(true);
7  });
8});
9
10describe("Resource Manager", () => {
11  it("should enforce memory limits", () => {
12    const manager = new WebGLResourceManager(gl);
13    expect(manager.createTexture(1024, 1024)).not.toBeNull();
14    expect(manager.createTexture(16384, 16384)).toBeNull(); // Exceeds limit
15  });
16});

6. Performance Patterns

6.1 Buffer Reuse

typescript
1// Bad - Creates new buffer every frame
2const buffer = gl.createBuffer();
3gl.bufferData(gl.ARRAY_BUFFER, data, gl.DYNAMIC_DRAW);
4gl.deleteBuffer(buffer);
5
6// Good - Reuse buffer, update only data
7gl.bufferSubData(gl.ARRAY_BUFFER, 0, data); // Update existing buffer

6.2 Draw Call Batching

typescript
1// Bad - One draw call per object
2objects.forEach(obj => {
3  gl.useProgram(obj.program)
4  gl.drawElements(...)
5})
6
7// Good - Batch by material/shader
8const batches = groupByMaterial(objects)
9batches.forEach(batch => {
10  gl.useProgram(batch.program)
11  batch.objects.forEach(obj => gl.drawElements(...))
12})

6.3 Texture Compression

typescript
1// Bad - Always uncompressed RGBA
2gl.texImage2D(gl.TEXTURE_2D, 0, gl.RGBA, gl.RGBA, gl.UNSIGNED_BYTE, image)
3
4// Good - Use compressed formats when available
5const ext = gl.getExtension('WEBGL_compressed_texture_s3tc')
6if (ext) gl.compressedTexImage2D(gl.TEXTURE_2D, 0, ext.COMPRESSED_RGBA_S3TC_DXT5_EXT, ...)

6.4 Instanced Rendering

typescript
1// Bad - Individual draw calls for particles
2particles.forEach((p) => {
3  gl.uniform3fv(uPosition, p.position);
4  gl.drawArrays(gl.TRIANGLES, 0, 6);
5});
6
7// Good - Single instanced draw call
8gl.drawArraysInstanced(gl.TRIANGLES, 0, 6, particles.length);

6.5 VAO Usage

typescript
1// Bad - Rebind attributes every frame
2gl.enableVertexAttribArray(0);
3gl.vertexAttribPointer(0, 3, gl.FLOAT, false, 0, 0);
4
5// Good - Use VAO to store attribute state
6const vao = gl.createVertexArray();
7gl.bindVertexArray(vao);
8// Set up once, then just bind VAO for rendering

7. Security Standards

7.1 Known Vulnerabilities

7.2 OWASP Top 10 Coverage

7.3 GPU Resource Protection

typescript
1// ✅ Implement resource limits
2const LIMITS = {
3  maxDrawCalls: 100,
4  maxTriangles: 1_000_000,
5  maxTextures: 32,
6  maxTextureSize: 4096,
7};
8
9function checkLimits(stats: RenderStats): boolean {
10  if (stats.drawCalls > LIMITS.maxDrawCalls) {
11    console.error("Draw call limit exceeded");
12    return false;
13  }
14  if (stats.triangles > LIMITS.maxTriangles) {
15    console.error("Triangle limit exceeded");
16    return false;
17  }
18  return true;
19}

8. Common Mistakes & Anti-Patterns

8.1 Critical Security Anti-Patterns

Never: Skip Context Loss Handling

typescript
1// ❌ DANGEROUS - App crashes on context loss
2const gl = canvas.getContext("webgl2");
3// No context loss handler!
4
5// ✅ SECURE - Handle gracefully
6canvas.addEventListener("webglcontextlost", handleLoss);
7canvas.addEventListener("webglcontextrestored", handleRestore);

Never: Unlimited Resource Allocation

typescript
1// ❌ DANGEROUS - GPU memory exhaustion
2for (let i = 0; i < userCount; i++) {
3  textures.push(gl.createTexture());
4}
5
6// ✅ SECURE - Enforce limits
7if (textureCount < MAX_TEXTURES) {
8  textures.push(gl.createTexture());
9}

8.2 Performance Anti-Patterns

Avoid: Excessive State Changes

typescript
1// ❌ BAD - Unbatched draw calls
2objects.forEach(obj => {
3  gl.useProgram(obj.program)
4  gl.bindTexture(gl.TEXTURE_2D, obj.texture)
5  gl.drawElements(...)
6})
7
8// ✅ GOOD - Batch by material
9batches.forEach(batch => {
10  gl.useProgram(batch.program)
11  gl.bindTexture(gl.TEXTURE_2D, batch.texture)
12  batch.objects.forEach(obj => gl.drawElements(...))
13})

9. Pre-Implementation Checklist

Phase 1: Before Writing Code

• Write failing tests for shaders, context, and resources
• Define performance budgets (draw calls <100, memory <256MB)
• Identify required WebGL extensions

Phase 2: During Implementation

• Context loss handling with recovery
• Resource limits and memory tracking
• Shader validation before compilation
• Use VAOs, batch draws, reuse buffers
• Instanced rendering for particles

Phase 3: Before Committing

• Tests pass: npm test -- --run tests/webgl/
• Type check: npm run typecheck
• Build: npm run build
• Performance verified (draws, memory)
• Fallback for no WebGL tested

10. Summary

WebGL provides GPU-accelerated graphics for JARVIS HUD. Key principles: handle context loss, enforce resource limits, validate shaders, track memory, batch draw calls, minimize state changes.

Remember: WebGL bypasses browser sandboxing - always protect against resource exhaustion. References: references/advanced-patterns.md, references/security-examples.md