Game Dev Mechanics: Toon Shading (Cel Shading) — How It Works

What Is Toon Shading?

Toon shading, also called cel shading, is a non-photorealistic rendering (NPR) technique that makes 3D objects look like hand-drawn cartoons. Instead of computing smooth, physically accurate light gradients across a surface, toon shading quantizes light intensity into a handful of discrete bands, producing flat regions of color with sharp transitions. Pair that with bold ink-style outlines, and you have a rendering style that has defined some of the most visually distinctive games ever made.

The name comes from cel animation, the traditional process of painting characters on transparent celluloid sheets layered over painted backgrounds. Each cel had only a few shading tones: a base color, one shadow tone, and maybe a highlight. Toon shading in real-time 3D replicates exactly this economy of tonal variation.

The Math Behind Toon Shading

Standard Diffuse Lighting

In conventional real-time rendering, diffuse lighting follows the Lambertian reflectance model. The brightness of a surface point depends on the angle between the surface normal $\vec{N}$ and the direction to the light source $\vec{L}$:

$$I_{diffuse} = \max(\vec{N} \cdot \vec{L},\; 0)$$

This dot product produces a smooth gradient: surfaces facing the light are bright, surfaces turned away are dark, with every value in between represented continuously.

Quantizing Into Bands

Toon shading takes that continuous intensity and snaps it to one of $B$ discrete levels:

$$I_{toon} = \frac{\lceil I_{diffuse} \times B \rceil}{B}$$

With $B = 3$, for example, every pixel on the surface can only be one of three brightness values. This is the core trick. A single line of shader code transforms photorealistic lighting into a cartoon look. In GLSL, it is simply:

float intensity = max(dot(N, L), 0.0);
intensity = ceil(intensity * uBands) / uBands;

Increasing $B$ produces subtler gradations (approaching smooth shading at high values), while $B = 2$ gives the starkest, most graphic look: just lit and shadow.

Specular Highlights: The Hard Cutoff

Photorealistic specular highlights (Phong or Blinn-Phong) produce soft, graduated bright spots. Toon shading replaces them with a binary on/off highlight using a step function:

$$S = \begin{cases} 1 & \text{if } (\vec{N} \cdot \vec{H})^{p} \geq t \\ 0 & \text{otherwise} \end{cases}$$

Here, $\vec{H}$ is the half-vector between the light and view directions, $p$ is the shininess exponent, and $t$ is the cutoff threshold. In GLSL, this is a single step() call. The result is a crisp, anime-style highlight rather than a blurry gradient.

Rim Lighting

Many toon shaders add a rim light, a bright edge glow that appears where the surface normal is nearly perpendicular to the view direction $\vec{V}$:

$$R = \left(1 - \max(\vec{N} \cdot \vec{V},\; 0)\right)^{e}$$

This technique, sometimes called Fresnel rim, mimics the backlighting common in anime and cartoon art. It separates characters from backgrounds and adds a sense of volume without breaking the flat-shaded aesthetic. The exponent $e$ controls the rim width; higher values produce a thinner, sharper rim.

Outline Rendering

The other signature element of toon shading is bold outlines around objects, mimicking ink lines in traditional animation. There are three primary techniques:

1. Inverted Hull Method

The most widely used approach renders each object twice:

• First pass: Render the toon-shaded object normally (front faces only)
• Second pass: Render the object again with front-face culling enabled, vertices extruded outward along their normals, colored solid black

The extruded back faces poke out around the silhouette of the front-facing geometry, creating a dark border:

$$\vec{p}_{outline} = \vec{p} + \vec{n} \cdot t$$

Where $t$ controls the outline thickness. This method is simple and GPU-friendly, and works on any geometry. Its main limitation is that it cannot produce interior detail lines (like the edge of a nose on a face), only silhouette outlines.

// Outline vertex shader
uniform float uThickness;
void main() {
    vec3 pos = position + normal * uThickness;
    gl_Position = projectionMatrix * modelViewMatrix * vec4(pos, 1.0);
}

// Outline fragment shader
void main() {
    gl_FragColor = vec4(0.0, 0.0, 0.0, 1.0);
}

2. Screen-Space Edge Detection

A post-processing approach that runs after the scene is rendered. It compares depth and/or normal values between neighboring pixels; large discontinuities indicate edges. This technique produces both silhouette and interior edges, but can struggle with thin geometry and requires a depth/normal pre-pass.

3. Geometry Shader Extrusion

A geometry shader examines each triangle's adjacency information to find silhouette edges (where a front-facing triangle meets a back-facing one) and generates line or quad geometry along those edges. This is precise but more expensive and requires adjacency data in the mesh.

In practice, the inverted hull method dominates game production due to its simplicity and performance. The demo above uses this technique.

Implementation Walkthrough

Step 1: Toon Material Shader

The vertex shader transforms normals into view space and passes the view-space position to the fragment shader:

varying vec3 vNormal;
varying vec3 vViewPosition;

void main() {
    vNormal = normalize(normalMatrix * normal);
    vec4 mvPos = modelViewMatrix * vec4(position, 1.0);
    vViewPosition = -mvPos.xyz;
    gl_Position = projectionMatrix * mvPos;
}

Step 2: Fragment Shader With Quantization

The fragment shader computes Lambertian diffuse, quantizes it, adds a hard specular highlight, and optionally applies a rim light:

uniform vec3 uColor;
uniform vec3 uLightDir;
uniform float uBands;

varying vec3 vNormal;
varying vec3 vViewPosition;

void main() {
    vec3 N = normalize(vNormal);
    vec3 L = normalize(uLightDir);
    vec3 V = normalize(vViewPosition);

    // Quantized diffuse
    float NdotL = dot(N, L);
    float intensity = max(NdotL, 0.0);
    intensity = ceil(intensity * uBands) / uBands;
    intensity = max(intensity, 0.12); // ambient floor

    // Hard specular highlight
    vec3 H = normalize(L + V);
    float spec = pow(max(dot(N, H), 0.0), 64.0);
    spec = step(0.5, spec);

    // Fresnel rim
    float rim = 1.0 - max(dot(N, V), 0.0);
    rim = smoothstep(0.55, 1.0, rim);

    vec3 color = uColor * intensity
               + vec3(1.0) * spec * 0.35
               + uColor * rim * 0.4;
    gl_FragColor = vec4(color, 1.0);
}

Step 3: Outline Pass

For each mesh in the scene, a second mesh is created with the outline material. The key settings are side: BackSide (cull front faces) and the vertex extrusion along normals:

const outlineMaterial = new THREE.ShaderMaterial({
    vertexShader: outlineVert,
    fragmentShader: outlineFrag,
    uniforms: {
        uThickness: { value: 0.03 }
    },
    side: THREE.BackSide
});

Step 4: Scene Assembly

Each visible object becomes a pair: the toon-shaded mesh and its outline clone. They share the same geometry and transform, but use different materials. Outline meshes should be rendered first (or use a render order) to avoid z-fighting artifacts.

const mesh = new THREE.Mesh(geometry, toonMaterial);
const outline = new THREE.Mesh(geometry, outlineMaterial);
outline.position.copy(mesh.position);
outline.rotation.copy(mesh.rotation);
scene.add(outline);
scene.add(mesh);

Advanced Techniques

Texture Ramps

Instead of mathematical quantization, some implementations sample a 1D ramp texture using $\vec{N} \cdot \vec{L}$ as the UV coordinate. This gives artists direct control over the exact color of each shading band, including hue shifts as well as brightness (warm highlights, cool shadows). Team Fortress 2 famously uses this approach, with hand-painted ramp textures per character that produce its distinctive illustrative look.

Normal Editing for Stylization

Guilty Gear Xrd pushed toon shading to its extreme by having artists manually edit vertex normals on character models. Because the shading depends entirely on the dot product $\vec{N} \cdot \vec{L}$, adjusting normals lets artists control exactly where shadow boundaries fall, producing results indistinguishable from hand-drawn 2D animation rendered in real-time 3D. This technique is labor-intensive but yields some of the most convincing results in real-time rendering.

Hatching and Cross-Hatching

For an illustrative or woodcut aesthetic, some NPR shaders replace solid color bands with procedural hatching patterns. Darker regions receive denser line patterns, while bright regions remain clean. This technique appears in games like Return of the Obra Dinn, which uses a 1-bit dithered rendering style inspired by early Macintosh graphics.

Games That Use Toon Shading

• The Legend of Zelda: The Wind Waker (2002) - Arguably the most iconic cel-shaded game, with bold outlines and two-tone shading that still holds up decades later
• Jet Set Radio (2000) - A pioneer of real-time cel shading in games, proving the technique could define an entire visual identity
• Borderlands series - Combines toon outlines with high-detail textures for a "graphic novel" hybrid style
• Guilty Gear Xrd / Strive - The most technically refined example of anime-style 3D rendering, using edited normals, controlled shadow shapes, and post-processing to mimic hand-drawn animation
• Ni no Kuni - Studio Ghibli-inspired cel shading for a storybook look that blends seamlessly with actual Ghibli cutscenes
• Okami (2006) - Sumi-e (ink wash painting) inspired rendering built on toon shading foundations
• Team Fortress 2 - Texture-ramp-based toon shading for its Pixar-meets-propaganda-poster art style
• Dragon Ball FighterZ - 3D characters rendered to look pixel-perfect like their 2D anime counterparts
• Splatoon series - Bright, colorful toon shading with thick outlines that reinforces its playful identity
• Hi-Fi Rush (2023) - A rhythm-action game with a striking comic-book cel-shaded style that pulses to the beat

Performance Considerations

Toon shading is generally cheaper than PBR since it involves simpler lighting math: no roughness maps, no environment probes, no IBL convolution. However, the outline pass doubles draw calls per object. Common optimizations include:

• Instanced outlines: Merge all outline geometry into a single instanced draw call
• Screen-space outlines: Replace per-object outline geometry with a single full-screen post-processing pass
• Distance-based outline scaling: Fade or thin outlines on distant objects to avoid visual noise
• Shared outline materials: Use a single outline material instance for all objects, updating only the thickness uniform

In practice, the performance overhead of toon shading is negligible on modern hardware. The real cost is in the art pipeline. Achieving a consistently appealing toon look requires careful attention to model topology, UV layout, and often hand-tuned normals.

Why Toon Shading Matters

Beyond aesthetics, toon shading teaches a foundational rendering principle: lighting is just data, and you can transform it however you want. The same dot product that drives photorealistic rendering can be quantized, remapped, or distorted to produce any visual style imaginable. This insight, that the rendering pipeline is a creative tool rather than just a physics simulator, makes non-photorealistic rendering one of the most flexible areas in real-time graphics.

For indie developers, toon shading offers a practical advantage too: it creates a visually distinctive game without requiring the massive texture, material, and lighting budgets that photorealistic rendering demands. A well-executed toon shader can make simple geometry look polished. Art direction matters more than polygon counts.

Try the interactive demo above to see these concepts in action. Drag to rotate the scene, and use the controls to adjust the number of shading bands, outline thickness, and rim lighting in real time.