Cel Shading / Toon Outline - Raymarched Scene with Hard Banding

Been working on a cel shading setup for a project and figured I'd share the standalone version I built to test the look. This one raymarches a simple scene (sphere on a plane) and then applies the toon shading on top - hard color banding, rim lighting, and an outline pass using surface normal edge detection.

The core idea is pretty straightforward: instead of smooth diffuse lighting, you quantize the dot(normal, lightDir) value into discrete steps. I'm using 4 bands here which gives a nice classic toon look, but you can bump it up or down depending on how stylized you want things. The outline detection works by checking how edge-on the surface normal is relative to the camera - when the dot product between the view direction and surface normal gets close to zero, you're looking at a silhouette edge.

One thing I spent way too long on was getting the rim light to play nice with the banding. If you just add rim on top of the quantized diffuse, it looks weird because you get this smooth gradient sitting on top of hard steps. So I ended up quantizing the rim contribution too, just with fewer bands (2 steps basically - on or off). Looks way more cohesive that way.

The light orbits around the scene so you can see how the bands move across the surface. I also threw in a subtle specular highlight that's been thresholded to give that classic anime hot-spot look.

precision mediump float;
uniform vec2 iResolution;
uniform float iTime;

// SDF for a sphere
float sdSphere(vec3 p, float r) {
    return length(p) - r;
}

// SDF for ground plane
float sdPlane(vec3 p, float h) {
    return p.y - h;
}

// Scene SDF - sphere sitting on a plane
float map(vec3 p) {
    float sphere = sdSphere(p - vec3(0.0, 0.5, 0.0), 0.8);
    float plane = sdPlane(p, -0.3);
    return min(sphere, plane);
}

// Calculate surface normal via gradient
vec3 calcNormal(vec3 p) {
    vec2 e = vec2(0.001, 0.0);
    return normalize(vec3(
        map(p + e.xyy) - map(p - e.xyy),
        map(p + e.yxy) - map(p - e.yxy),
        map(p + e.yyx) - map(p - e.yyx)
    ));
}

// Raymarching
float raymarch(vec3 ro, vec3 rd) {
    float t = 0.0;
    for (int i = 0; i < 80; i++) {
        vec3 p = ro + rd * t;
        float d = map(p);
        if (d < 0.001) break;
        if (t > 20.0) break;
        t += d;
    }
    return t;
}

// Soft shadow for ground plane
float softShadow(vec3 ro, vec3 rd, float mint, float maxt) {
    float res = 1.0;
    float t = mint;
    for (int i = 0; i < 40; i++) {
        float h = map(ro + rd * t);
        res = min(res, 8.0 * h / t);
        t += clamp(h, 0.02, 0.1);
        if (h < 0.001 || t > maxt) break;
    }
    return clamp(res, 0.0, 1.0);
}

// Quantize value into discrete bands for toon look
float toonBand(float val, float bands) {
    return floor(val * bands) / bands;
}

void main() {
    vec2 uv = (gl_FragCoord.xy - 0.5 * iResolution.xy) / iResolution.y;

    // Camera setup
    vec3 ro = vec3(0.0, 1.2, 3.5);
    vec3 target = vec3(0.0, 0.2, 0.0);
    vec3 fwd = normalize(target - ro);
    vec3 right = normalize(cross(fwd, vec3(0.0, 1.0, 0.0)));
    vec3 up = cross(right, fwd);
    vec3 rd = normalize(fwd * 1.5 + right * uv.x + up * uv.y);

    // Orbiting light
    vec3 lightPos = vec3(3.0 * cos(iTime * 0.7), 3.0, 3.0 * sin(iTime * 0.7));

    // Background gradient - toon sky
    vec3 skyTop = vec3(0.35, 0.55, 0.85);
    vec3 skyBot = vec3(0.7, 0.8, 0.92);
    float skyGrad = toonBand(clamp(rd.y * 0.5 + 0.5, 0.0, 1.0), 3.0);
    vec3 col = mix(skyBot, skyTop, skyGrad);

    float t = raymarch(ro, rd);

    if (t < 20.0) {
        vec3 p = ro + rd * t;
        vec3 n = calcNormal(p);
        vec3 lightDir = normalize(lightPos - p);
        vec3 viewDir = normalize(ro - p);
        vec3 halfDir = normalize(lightDir + viewDir);

        // Material colors
        vec3 matColor;
        float isSphere = step(sdSphere(p - vec3(0.0, 0.5, 0.0), 0.81), 0.0);

        if (isSphere > 0.5) {
            matColor = vec3(0.85, 0.25, 0.2); // Red sphere
        } else {
            // Checkerboard on ground, also quantized
            float checker = mod(floor(p.x * 2.0) + floor(p.z * 2.0), 2.0);
            matColor = mix(vec3(0.55, 0.7, 0.55), vec3(0.35, 0.5, 0.35), checker);
        }

        // Diffuse with toon banding (4 steps)
        float diff = max(dot(n, lightDir), 0.0);
        float toonDiff = toonBand(diff, 4.0);
        // Bump up minimum so shadow band isn't pure black
        toonDiff = toonDiff * 0.7 + 0.3;

        // Specular - thresholded for anime hot-spot
        float spec = pow(max(dot(n, halfDir), 0.0), 32.0);
        float toonSpec = step(0.5, spec); // binary on/off specular

        // Rim lighting - quantized to on/off
        float rim = 1.0 - max(dot(viewDir, n), 0.0);
        rim = pow(rim, 3.0);
        float toonRim = step(0.55, rim);

        // Shadow
        float shadow = softShadow(p + n * 0.01, lightDir, 0.02, 5.0);
        float toonShadow = step(0.5, shadow); // hard shadow edge

        // Combine
        col = matColor * toonDiff * mix(0.5, 1.0, toonShadow);
        col += vec3(1.0, 0.95, 0.9) * toonSpec * 0.6; // specular highlight
        col += vec3(0.4, 0.5, 0.7) * toonRim * 0.3; // rim tint

        // Outline detection via normal-view angle
        float edge = dot(viewDir, n);
        float outline = 1.0 - smoothstep(0.15, 0.25, edge);
        col = mix(col, vec3(0.05, 0.02, 0.05), outline);

        // Distance-based outline thickness adjustment
        // Edges get thinner as they go further from camera
        float distFade = smoothstep(8.0, 2.0, t);
        col = mix(col, mix(col, vec3(0.05, 0.02, 0.05), outline), distFade);
    }

    // Simple vignette
    vec2 vUv = gl_FragCoord.xy / iResolution.xy;
    float vignette = 1.0 - 0.3 * length(vUv - 0.5);
    col *= vignette;

    // Gamma
    col = pow(col, vec3(1.0 / 2.2));

    gl_FragColor = vec4(col, 1.0);
}

The band count is the main thing to tweak if you want a different feel. 2-3 bands gives a more extreme flat look (think Jet Set Radio), 4-5 is more like Guilty Gear or Dragon Ball FighterZ territory, and above that you start losing the toon effect and it just looks like slightly chunky smooth shading.

One thing I'm still not totally happy with is the outline approach. The normal-based edge detection works ok for silhouettes but misses creases and hard edges between different objects. For a real game you'd probably want a sobel filter on the depth buffer or do the inverted hull trick with backface geometry. Has anyone here implemented depth-based outlines in a fullscreen pass that actually looks good? I keep getting artifacts at depth discontinuities and I'm wondering if there's a standard workaround I'm missing.