Project 2

Shadow Mapping

Shadow mapping for directional and spot lights on the deferred pipeline, from binary shadows through PCF to variance and moment shadow maps blurred in compute shaders.

Course
CS 562, Advanced Real Time Rendering
Submitted
1 March 2026
Status
Complete

Project overview

This project adds a shadow mapping system to the deferred rendering pipeline from Project 1. It supports several shadow filtering algorithms, from basic binary shadows to Variance Shadow Mapping (VSM) and Moment Shadow Mapping (MSM), for smooth soft shadows with few artifacts.


Architecture overview

Updated rendering pipeline

The pipeline now includes passes for shadow depth generation and moment blurring:

═════════════════════════════════════════════════════════════
                    RENDERING PIPELINE
═════════════════════════════════════════════════════════════

1. GEOMETRY PASS
   ↓
   Render scene geometry to G-Buffer

2. SHADOW PASS
   ↓
   Render scene from light's perspective to Shadow FBO
   Calculates and stores up to 4 depth moments

3. MOMENT BLUR PASS (Compute Shaders)
   ↓
   Apply Gaussian blur to shadow moments for VSM/MSM

4. LIGHTING PASS (Deferred)
   ↓
   Sample G-Buffer & Shadow Maps to calculate final color
   Implements Direct, PCF, VSM, or MSM filtering

═════════════════════════════════════════════════════════════

Shadow buffer setup

The shadow pass uses a 32-bit float framebuffer to store depth information and moments:

TextureFormatContainsUse
shadowMapRGBA32FDepth, Depth², Depth³, Depth⁴Shadow calculations, VSM/MSM
blurredShadowMapRGBA32FBlurred momentsSoft shadow filtering

Scene setup

The scene is an interior environment built around complex shadow interactions: contact hardening, overlapping shadows, and fine-geometry detail. The assets were picked to test different shadow interactions:

  • Central desk and seating
    • Wooden desk: holds four objects, a bronze lion head statue, a concrete cat, a marble bust, and a carved wooden elephant. Their varying geometric complexity tests detailed self-shadowing and contact shadows.
    • Lounge seating: a brown leather mid-century lounge chair and modern black armchairs. Their curved surfaces and overlapping geometry test soft shadow transitions and nested occlusion.
  • Electronics and storage
    • Technical gear: a vintage video camera, a cassette tape player, and a game console. Protruding parts (lenses, buttons) test whether the shadow map captures fine, thin details without “peter-panning.”
    • Academic assets: an encyclopedia set and a drawer unit. The repetitive, tight gaps between books were chosen to detect shadow aliasing and “shadow acne” in uniform structures.
  • Traditional tea set
    • Furniture: a low Chinese tea table and high-back armchairs.
    • Porcelain set: a teapot, sugar pot, and cups. These small-scale objects cast sharp contact shadows, testing precision at close-range light distances.
  • Cabinet and foliage
    • Slatted cabinet: a horizontal unit with vertical slats, which casts a repeating shadow pattern and tests filtering consistency across regular intervals.
    • Potted plants: two plants whose organic foliage gives high-frequency shadow patterns, used to evaluate how PCF, VSM, and MSM handle complex geometry.
  • Environment
    • Floor: a wooden plank floor, the primary large-scale receiver, where shadow stretching, projection distortion, and penumbra softening are visible.
    • Atmosphere: an HDR skybox for clean ambient lighting and a soft horizon, which isolates the shadow silhouettes.
Click to zoom
| Rotating around scene with comparison between No Shadow / Binary Shadows |

Light types and shadow projections

Shadow mapping works with both linear and non-linear projections, depending on the light source:

Directional light (orthographic projection)

Directional lights simulate distant sources like the sun, where light rays are parallel.

  • Projection: an orthographic matrix.
  • Depth metric: standard depth ([0, 1] range) derived from the camera’s gl_FragCoord.z.
  • Properties: a configurable orthoSize defines the shadow volume.
Click to zoom

Spot light (perspective projection)

Spot lights simulate local point sources with a specific direction and cone angle.

  • Projection: a perspective matrix.
  • Depth metric: linearized depth, calculated as (position.w - nearPlane) / (farPlane - nearPlane).
  • Properties: a configurable fovAngle controls the light cone and shadow resolution density.
Click to zoom

Projection comparison: shadow stretching

For the same light position and shadow map resolution, orthographic projection typically yields “tighter” shadows across a large scene than perspective projection.

  • Consistent shadows (ortho): because orthographic projection uses parallel light rays, the shadow of every object keeps a consistent angle and shape, regardless of its position relative to the light’s center axis.
  • Stretched shadows (perspective): in perspective projection, light rays radiate from a single point, so shadows for objects near the edges of the light’s view are stretched more than those in the center, which can make the shadows look less “tight” across the scene.
Click to zoom
| Shadow difference between Ortho vs perspective light |

Frustum fitting and resolution quality

Shadow quality depends on how much of the shadow map resolution lands on the geometry that needs it.

  • Ortho size (directional): the orthoSize parameter controls the dimensions of the orthographic projection box. A smaller orthoSize concentrates the whole shadow map resolution into a smaller volume, giving sharper shadows. A large orthoSize forces the fixed shadow map resolution to cover a larger surface area, which increases aliasing and shows “pixelated” stair-stepping on shadow edges.
  • Frustum size (spot/perspective): for spot lights, increasing the fovAngle or the distance between the near/far planes expands the shadow volume. As with orthographic projection, this forces the fixed shadow map resolution to cover a larger world-space area, which increases aliasing and loses fine shadow detail.
  • Optimal fitting: for the best shadow quality at a given depth texture size, the light’s frustum should be scaled to tightly encapsulate only the necessary geometry. This maximizes texel density and minimizes aliasing artifacts without requiring larger textures.
Click to zoom

Shadow map visualization

The debug shader has modes for the shadow data the lighting pass reads.

Raw shadow map

This mode projects the shadow depth texture onto the scene from the camera’s perspective, showing the raw depth data stored in the shadow map.

  • Technical mapping:
    • The RGBA channels are mapped to (z, z^2, z^3, z^4), the four depth moments.
    • Color ramp: as depth (z) increases from 0 to 1, the visualization goes from a darkish red (at low z values) to pure white (at the far plane).
    • Moment tinting: because the higher moments (z^2, z^3, z^4) decrease in value much faster than z itself for values less than 1, the red channel dominates the color at shorter distances, giving a warm/reddish gradient that only converges to grayscale/white as z approaches 1.
  • Observations:
    • Depth gradient: when moving the light direction toward the camera’s viewpoint, front-most objects show dark reddish tones while distant objects fade to white. The color gradient shifts across the scene as the light moves, following the changing depth values in the shadow map.
    • Cross-projection consistency: switching between orthographic and perspective light sources gives identical depth values, so the depth linearization and light-space math agree across both projection types.
Click to zoom
Click to zoom

Blurred shadow map

This mode shows the result of the two-pass Gaussian compute shader blur applied to the shadow moments.

  • Visual impact:
    • Macro view: at a distance, the blurred shadow map looks nearly identical to the raw map, so the overall depth structure is preserved.
    • Micro view (zoomed in): closer in, the gradients are much smoother. Aliasing artifacts on sharp edges (chairs and tables) resolve into soft, continuous transitions.
    • Leaf geometry: organic structures like leaves, which show high-frequency “checkerboard” aliasing in the raw map, come out as smooth shadows in the blurred version.
  • Blur radius control: adjusting the blur radius at runtime trades shadow softness against geometric detail. Increasing the radius diffuses the shadows further, giving wider penumbras for the VSM/MSM filters.
Click to zoom

Shadow algorithm implementations

Direct binary shadows

The baseline implementation performs a direct depth comparison between the current fragment’s light-space depth and the recorded shadow map depth to determine visibility (0.0 or 1.0).

  • Shadow acne mitigation: two layers of bias prevent self-shadowing artifacts (shadow acne):

    • Normal bias (world space): offsets the world position along the surface normal: biasedPos = worldPos + normal * normalBias.
    • Depth bias (light space): a slope-dependent offset applied during comparison:
        bias = depthBias * (1.0 - dot(normal, lightDir))
        float biasedPixelDepth = pixelDepth - bias; 
  • Shader implementation (deferred_lighting.frag L334-338):

    if (shadowMapMethod == 0) {
        if (lightDepth < (biasedPixelDepth)) {
            inShadow = true;
        }              
    }
  • Visual analysis:

    • Macro fidelity: at a distance, the scene is shadowed correctly with stable silhouettes. Moving the light shows objects casting shadows onto one another while surface colors are preserved.
    • Aliasing artifacts: up close, aliasing and jaggedness become apparent, most of all on curved geometry (chair edges) and high-frequency details (potted plant foliage), which look pixelated because there is no hardware filtering or sub-texel sampling.
Click to zoom

Percentage-Closer Filtering (PCF)

PCF samples a local neighborhood of the shadow map and averages the visibility results, which softens the harsh stair-stepping of binary shadows.

  • Technical implementation:

    • Stochastic sampling: a 16-sample Poisson disk pattern. This stochastic distribution turns the regular banding artifacts of a grid into less noticeable noise, giving a more natural penumbra.
    • Mathematical model: the shadow factor is the average of multiple binary depth tests: shadowFactor = Sum(SampledDepth < BiasedDepth ? 1.0 : 0.0) / 16.0
  • Shader implementation (deferred_lighting.frag L339-357):

    else if (shadowMapMethod == 1) {
        // --- PCF with Poisson Disk Sampling ---            
        vec2 texelSize = 1.0 / textureSize(shadowMap, 0).xy;
        float shadowAccumulator = 0.0;                        
    
        for (int i = 0; i < 16; i++) {
            float sampledDepth = texture(shadowMap, shadowIndex + poissonDisk[i] * texelSize * 2).r;
            // Binary test for each sample
            if (sampledDepth < (biasedPixelDepth)) {
                shadowAccumulator += 1.0;
            }
        }
        
        shadowFactor = shadowAccumulator / 16.0;             
        visibility = 1.0 - shadowFactor;
    }
  • Visual analysis:

    • Fidelity improvements: PCF looks much smoother than binary shadows and tames aliasing across the scene:

      • Structural silhouettes: edges of chairs and tables are more continuous and smooth.

        Click to zoom
      • Fine occlusion: small-scale self-occlusion details, such as the areas around the eyes of the marble bust, are much clearer.

        Click to zoom
      • Complex organic detail: the shadows of plant leaves are smoothed out and soft, without the jagged edges from before.

        Click to zoom
      • Curved geometry: the shadow curve cast onto the wooden cabinet has a clean gradient.

    • Limitations: PCF resolves aliasing, but it can introduce a “hazy” or “smeared” look. Because the filter applies a uniform average, it can lose the crisp definition required for sharp contact shadows, and it may show slight noise depending on the sample count.


Variance Shadow Mapping (VSM)

VSM treats depth as a statistical distribution, using the first two moments (mean and mean-squared depth) to estimate the probability that a fragment is in shadow.

  • Comparison to PCF: VSM shadows are smoother than standard PCF and keep more definition with less “smear.” Because VSM relies on linear combinations of moments, the shadow map can be pre-filtered in compute shaders, which improves performance for soft shadows.

    Click to zoom
  • Blur radius and aliasing trade-off: The blur radius can be changed at runtime, which balances shadow crispness against aliasing:

    • Lower blur radius: shadows are sharper and more defined, but minor aliasing artifacts may remain visible.
    • Higher blur radius: as the radius increases, shadows smooth out, removing aliasing at the cost of the shadow’s definition.
    Click to zoom
  • Light bleeding (the Chebyshev limitation): VSM fails on light bleeding, particularly with layered shadows (one geometry casting a shadow over another shadow, or a single complex geometry casting nested shadows). In these cases, VSM estimates visibility incorrectly, so light “leaks” around the edges of the higher geometry.

    • Example, chair geometry: a faint light edge is visible in the shadows of the chair armrests (hands), where the shadow should be solid.
    • Example, marble bust: a faint outline of the bust is visible within the shadow it casts onto the table, a “ghosting” effect.
    • Reasoning: VSM uses Chebyshev’s inequality to estimate visibility probability, which is not accurate enough for overlapping depth distributions.
    Click to zoom
  • Mathematical estimator (Chebyshev’s inequality): The visibility probability is estimated from the variance (sigma^2) and the distance of the fragment depth (d) from the mean depth (M1):

    • Variance: sigma^2 = M2 - (M1)^2
    • Chebyshev probability: P_max = sigma^2 / (sigma^2 + (d - M1)^2)
    • Final visibility: if d <= M1, visibility is 1.0; otherwise, visibility is P_max.
  • Shader implementation (deferred_lighting.frag L330-372):

    // Depth and Bias Calculation (L330-331)
    float bias = computeBias(shadowCoord, N, L);                
    float biasedPixelDepth = pixelDepth - bias;            
    
    // VSM Implementation (L358-372)
    else if (shadowMapMethod == 2) {
        float m1 = texture(blurredShadowMap, shadowIndex).r;
        float m2 = texture(blurredShadowMap, shadowIndex).g;
    
        if(biasedPixelDepth <= m1) {
            visibility = 1.0f;
        } else {
            float mu = m1;
            float sigma2 = m2 - (m1 * m1);
    
            visibility = sigma2 / (sigma2 + pow((pixelDepth - mu), 2));
        }
    }

Moment Shadow Mapping (MSM)

MSM uses four depth moments (z^1, z^2, z^3, z^4) to create soft shadows.

  • Improvements over VSM (reduced light bleeding): Switching from VSM to MSM reduces the light bleeding artifacts in complex overlapping geometry. The shadows stay well defined, smooth, and soft, and the layered occlusion where VSM leaked light is handled correctly.

    Click to zoom
  • Moment processing and solver: Like VSM, MSM supports pre-filtering by storing all four moments in a single RGBA32F texture and blurring them in a compute shader. To determine visibility, the implementation uses a 3x3 Cholesky decomposition to solve the moment problem, which gives a much tighter upper bound for the visibility function than Chebyshev’s inequality.

  • The “thin contact shadow” problem: MSM reduces light leaking, but it introduces thin or disappearing contact shadows. Because it relies on higher-order depth moments combined with Cholesky decomposition, geometry that occupies a very small space in the shadow map (the base of the chairs, the elephant’s feet) casts shadows that are extremely faint or virtually invisible.

  • Tuning factors and calibration: Several tuning parameters adjust the moment computation to recover these thin shadows:

    • Moment scale: pre-scales the moments before computing the visibility probability: moments *= momentShadow.scale;.

    • Alpha and moment mixing: generates a mix factor during moment computation: float alpha = 0.01f / (pow(10.0, momentShadow.alpha));.

      • Impact: scaling down alpha generates much thicker contact shadows for small geometry (visible on table edges, chair legs, and the elephant’s feet).
      • Limitation: if alpha is pushed below a certain threshold, it introduces edge artifacts, where shadows at boundaries start bleeding light again at grazing angles.
      Click to zoom
    • Bleed reduction (thresholding): remaps the visibility probability to use a minimum threshold (p):

      float p = momentShadow.bleedReduction; 
      visibility = clamp((visibility - p) / (1.0 - p), 0.0, 1.0);
      • Strategy: combining the alpha slider with bleed reduction thickens contact shadows (making the teapot’s shadow explicitly visible) while avoiding the light bleeding edge artifacts.
      Click to zoom
    • Visibility scale: applies a post-process power curve to the final computed visibility: visibility = pow(visibility, momentShadow.visScale);.

      • Impact: scaling visibility up indiscriminately makes shadows “thicker than expected” across the scene.
      Click to zoom
  • Direct scaled moments mode: A toggle bypasses the alpha mix() calculation entirely, using the downscaled moments directly with a minor bleed reduction factor, instead of balancing the sliders against each other.

    • Advantage: this achieves the same thickened contact shadows natively, and it does not suffer from the edge-bleeding artifacts at grazing light angles that occur when downscaling the alpha value.
    Click to zoom
  • Blur radius trade-off: As with VSM, MSM is tightly coupled to the compute shader blur:

    • A lower blur radius gives crisp definition for small geometry like the teapot, but leaves aliased, jagged shadows on the macro geometry (table/chair).
    • An increased blur radius gives smooth shadows for the macro geometry, but the small teapot shadows lose definition entirely.
    • With increased blur radius (above 30), we can also see some of the layered shadows (top leaves casting shadows on bottom leaves) disappear.
    • With current thread size, max support for blur radius is limited to 64, increasing thread size (e.g : 256) helps support higher blur radius like 100.
    Click to zoom
    | Blur radius modulates the transition from detailed shadows to less detailed shadows |
    Click to zoom
    | Lower blur radius (4-5) preserve fine detail in leaf geometry with minimal softening, while increased blur (40-50) progressively degrades geometric detail |
    Click to zoom
    | Lower blur radius maintains shadow definition and geometric acuity on the teapot; excessive blur causes near-complete attenuation of shadow detail |
  • Shader implementation (deferred_lighting.frag L111-174, L373-385):

    // Cholesky solver and Moment calculation logic
    float computeMomentShadow(vec4 moments, float zf, vec3 N, vec3 L) {
        // ... Moment evaluation logic ...
        float D = max(0.0, (c2 * c2) - (4.0 * c3 * c1 ));
        float Z2 = (-c2 + sqrt( D )) / (2 * c3);
        float Z3 = (-c2 - sqrt( D )) / (2 * c3);
        // ... (Refer to source for full implementation)
    }
    
    // Main Selection logic (L373-385)
    else if (shadowMapMethod == 3) {                               
        vec4 moments = texture(blurredShadowMap, shadowIndex);                
        moments *= momentShadow.scale;
    
        visibility = computeMomentShadow(moments, biasedPixelDepth, N, L);
        visibility = (1.0 - visibility);
    
        float p = momentShadow.bleedReduction;
        visibility = clamp((visibility - p) / (1.0 - p), 0.0, 1.0);
        visibility = pow(visibility, momentShadow.visScale);                 
    }

Integration with final color

The calculated visibility (or the boolean inShadow state) modulates the contribution of the light source in the final deferred pass. The ambient component stays constant to prevent pitch-black areas, and the diffuse and specular (PBR) contributions are scaled by the visibility factor.

  • Shader implementation (deferred_lighting.frag L392-400):

    // --- Final Color Calculation ---
    vec3 ambient = vec3(0.03) * albedo;        
    vec3 color = vec3(0.0f);
    
    if (shadowMapMethod == 0) {
        // Binary Shadow: Use boolean state
        color = inShadow ? ambient : (ambient + Lo);    
    } else {
        // Filtered Shadow (PCF/VSM/MSM): Use continuous visibility factor
        color = ambient + (visibility * Lo);
    }
    • Lo: the full PBR lighting contribution (diffuse + specular) calculated in CalculatePBRLight.
    • Visibility: a range from 0.0 (fully shadowed) to 1.0 (fully lit), which gives smooth transitions in soft shadow penumbras.

A final walk around the scene with MSM shadows and light movement:

Click to zoom
Click to zoom
Click to zoom

Blurring and compute shaders

VSM and MSM need the shadow moments blurred. The blur is a two-pass Gaussian in an OpenGL compute shader (shadowblur.comp). Splitting the 2D blur into separate horizontal and vertical passes reduces the algorithmic complexity and improves baseline performance.

Technical implementation and optimizations

  • Two-pass layouts: The compute shader compiles different group layouts depending on pass direction, for parallel memory coalescing:
    • Horizontal pass (#define HORIZONTAL): dispatched with a local workgroup size of (128, 1, 1). Samples the raw shadow map and writes to an intermediate image.
    • Vertical pass: dispatched with a local workgroup size of (1, 128, 1). Samples the intermediate image and writes the final blurred outcome.
  • Shared memory caching (shared vec4 v[128+101];): Instead of every thread reading all overlapping samples directly from the global texture, which is slow, each thread group pre-loads a full tile of image data into the faster GPU shared memory cache.
  • Thread synchronization (memoryBarrierShared() / barrier()): Before calculating the blur sum, the threads halt and synchronize so the entire local cache array v is populated by all participating threads. Then the threads traverse the shared memory to calculate the final pixel moments.
  • Boundary handling and halo prevention: The blur radius extends past the edges of the local workgroup thread pool, so specific threads (where lpos < blurWidth) fetch the extra “apron” pixels. Out-of-bounds sampling at the extreme edges of the screen is restricted with clamp(..., vec2(0), imgSize - 1).
  • SSBO Gaussian weights: The dynamically calculated Gaussian weights are streamed to the shader through a read-only Shader Storage Buffer Object (SSBO: layout (std430, binding = 0)). This avoids the CPU-GPU overhead of uploading uniform arrays every frame and still allows the blur kernel size to change.

UI and debug visualization

The application uses ImGui for runtime control, calibration, and debugging of the scene rendering and shadow systems.

Interactive controls

  • Shadow map method: switch between the No Shadows, Binary, PCF, VSM, and MSM implementations.
  • VSM/MSM blur radius: tune the Gaussian compute shader radius at runtime (1-20), which sets the soft shadow penumbra size.
  • Lighting controls:
    • Toggle between a directional light and a spot light.
    • Adjust global light direction, color, and intensity.
  • Global shadow settings: tune global Exposure, Normal Bias, and Depth Bias to mitigate shadow acne across all methods.
  • Moment shadow (MSM) settings: selecting MSM enables extra tuning sliders:
    • Alpha: tune the mixing factor used during moment reconstruction.
    • Moments scale: pre-scale the raw moments fetched from the G-Buffer.
    • Visibility scale: apply a post-process scaling curve to the final calculated visibility.
    • Light bleed reduction: adjust the minimum probability threshold to prevent leaking.
    • Direct scaled moments toggle: bypass the alpha mix logic to prioritize contact shadow thickness over edge bleeding.

Debug visualization modes

The debug shader (deferred_lighting_debug.frag) has views for checking that the G-Buffer and render targets are populated correctly before lighting is applied:

  • Material and geometry buffers: World Position, Normal, Albedo, Metallic, and Roughness data.
  • Shadow pipeline buffers:
    • Shadow map: the raw linearized/normalized depth and computed moment textures.
    • Blurred shadow map: the result of the compute shader Gaussian blur pass applied to the raw moments.

File structure

src/
├── shaders/
│   ├── shadow.vert              # Shadow pass vertex transformations
│   ├── shadow.frag              # Moment calculation (depth¹ to depth⁴)
│   ├── shadowblur.comp          # Gaussian blur compute shader
│   ├── deferred_lighting.frag   # Main PBR shader with shadow filtering logic
│   └── deferred_lighting_debug.frag # Visualization of G-Buffer and Shadow Maps
│
├── include/
│   ├── ShadowMapBlur.h          # Compute shader blur manager
│   ├── DirectionalLight.h       # Directional light & shadow matrices
│   ├── SpotLight.h              # Spot light & shadow matrices
│   └── FrameBuffer.h            # Updated for Shadow FBO support
│
├── ShadowMapBlur.cpp            # Gaussian weight generation & dispatch logic
├── DirectionalLight.cpp         # Light-space matrix calculations
└── Scene.cpp                    # Orchestrates geometry, shadow, and blur passes

Conclusion

What the shadow mapping pipeline ended up with:

  • Moment-domain filtering: shadow filtering moved to the moment domain (VSM/MSM) to compute soft shadows smoothly.
  • Compute shaders: a two-pass Gaussian blur on the shadow moments.
  • Soft shadows: light bleeding and shadow acne are mitigated across complex geometry.
  • Performance: the deferred shading architecture’s performance benefits are kept, with dynamic soft penumbras added to the scene.

References