Stencil buffer tricks for shader effects — what are you actually using it for?
I've been digging into stencil buffer usage beyond the obvious "outline selected object" case and honestly I feel like it's one of the most underutilized tools in real-time shader work. Wanted to open a discussion rather than post another tutorial because I'm curious what approaches people have actually shipped.
My current use case: I'm rendering a portal effect where geometry on the other side of the portal should only be visible through the portal opening. The stencil pass writes a mask for the portal quad, then the scene behind it renders with stencil test enabled. Straightforward in theory, but I ran into issues with the fragment shader needing to reconstruct depth correctly so objects behind the portal don't clip with the floor geometry in the main scene. My current depth hack feels wrong:
// Reconstructing world position from depth for portal scene
vec3 reconstructPos(float depth, vec2 uv) {
float z = depth * 2.0 - 1.0;
vec4 clipPos = vec4(uv * 2.0 - 1.0, z, 1.0);
vec4 viewPos = uInvProj * clipPos;
viewPos /= viewPos.w;
vec4 worldPos = uInvView * viewPos;
return worldPos.xyz;
}
Is this the right way to handle it, or is there a cleaner depth buffer write strategy?
Beyond portals — I've also seen stencil used for:
- Wet surface masking (only parts of a surface touched by rain get the wet normal map)
- Decal layering priority so decals don't bleed across material boundaries
- Subsurface scattering approximations that need a skin-only mask pass
What I'm really curious about is whether anyone has used stencil in a deferred rendering pipeline. I'm moving from forward to deferred and I'm not sure which of these tricks survive the transition cleanly. The wet surface one in particular feels like it might need a full G-buffer slot instead of stencil, which seems wasteful.
Also: any gotchas with stencil on mobile GPUs? I've heard some tile-based architectures handle stencil reads really differently and it can murder your fill rate if you're not careful.