Learning Late, Guiding Early: Timestep-Decoupled Semantic Guidance for Fair Face Generation

The gist: Semantic Boundary Predictor (SBP) introduces an inference-time framework that performs demographic guidance through a one-shot intervention during reverse denoising, learning linear semantic boundaries from late-stage latent representations to achieve fair face generation without model retraining. This method effectively reduces fairness disparity across various demographic attributes while maintaining perceptual image quality.

Motivation and Problem

Demographic imbalance in synthetic face generation can propagate to downstream face recognition systems, making fairness an important consideration when diffusion models are used for data generation <ref:2608.25862#pg2> Existing fairness-aware generation approaches often require model retraining, architectural modifications, or repeated guidance throughout the reverse diffusion process <ref:2608.25862#pg4> This work is motivated by the observation that latent representations at different diffusion timesteps play distinct semantic roles: late-stage latents provide stronger demographic separability, whereas early-stage latents offer greater flexibility for semantic intervention <ref:2608.25862#pg2> The analysis shows that latent representations at different timesteps exhibit different semantic characteristics, which indicates that the reverse denoising process is not equally informative for fairness-aware guidance at every stage <ref:2608.25862#pg4>.

Proposed Framework: Semantic Boundary Predictor (SBP)

The framework decouples semantic boundary learning from semantic guidance across diffusion timesteps <ref:2608.25862#pg2> SBP learns semantic boundaries from discriminative late-stage latent representations and applies them during the early denoising stage, where guidance is applied only once before the remainder of the reverse diffusion process proceeds unchanged <ref:2608.25862#pg4>. The SBP model estimates a semantic decision boundary for each demographic attribute, modeled by a linear classifier f(z0) = b T z0 + c <ref:2608.25862#pg4>. During inference, the learned semantic boundary is applied as ZT ← ZT ± δb (2) where b is the unit normal vector learned by SBP and δ ∈ Z≥0 controls the strength of the semantic guidance <ref:2608.25862#pg4>.

Methodology and Analysis

The method involves several key steps, including:

1. Generating a synthetic dataset by sampling facial images with the pre-trained LDM and then predicting their demographic attributes <ref:2608.25862#pg4>.

2. Extracting the corresponding latent representations in the final denoising timestep (t = 0), where semantic information is most discriminative <ref:2608.25862#pg4>.

3. Training an independent Semantic Boundary Predictor (SBP) for each demographic attribute, implemented as a lightweight linear classifier operating on a compact-subspace variant of SBP that performs semantic boundary learning and guidance in a PCA subspace <ref:2608.25862#pg4>.

The timestep-decoupled strategy is based on the hypothesis that coarse semantic structure is preserved along the diffusion trajectory, allowing directional biases introduced during the initializing step to affect denoising in successive timesteps <ref:2608.25862#pg4>. The analysis confirms that prediction accuracy consistently improves as denoising progresses, indicating that latent representations become increasingly linearly separable during the reverse denoising process <ref:2608.25862#pg4>. Furthermore, latent representations at t = 0 exhibit strong semantic separability and require far fewer principal components to preserve dominant variance than early-stage representations <ref:2608.25862#pg4>.

Experimental Results

Experiments on CelebA-HQ and FFHQ demonstrate substantial improvements in demographic fairness, reducing fairness disparity by 98% for gender, 95% for binary race, and 15% for four-class race <ref:2608.25862#pg4>. The proposed approach improves demographic fairness while maintaining perceptually high-quality facial image generation <ref:2608.25862#pg4>. For gender generation, SBP reduces the fairness disparity from 0.051 to 0.001, corresponding to an approximately 98% reduction <ref:2608.25862#pg4>. In the four-class race setting, SBP achieves a 15% reduction in fairness disparity while preserving perceptual quality <ref:2608.25862#pg4>. Multi-attribute generation shows that SBP achieves the lowest fairness disparity (FD = 0.055) for joint gender–race generation compared to existing inference-time baseline methods <ref:2608.25862#pg4>.

Conclusion and Implications

SBP enables fairness-aware generation without retraining, fine-tuning, or requiring external balanced datasets <ref:2608.25862#pg4>. The results confirm that a simple semantic boundary predictor learned from latent representations from late-stage is sufficient for effective inference-time demographic control <ref:2608.25862#pg4>. SBP offers a practical approach for generating balanced synthetic facial data, supporting the development of fairer downstream learning systems without retraining existing latent diffusion models <ref:2608.25862#pg4>. The modest additional inference cost suggests that the fair generation improvements require only a modest additional inference cost <ref:2608.25862#pg4>.

Limitations and Future Work

The proposed SBP learns a separate semantic boundary for each demographic attribute, which requires training and maintaining multiple SBPs for supporting additional attributes <ref:2608.25862#pg4>. Furthermore, boundary learning depends on external attribute classifiers, so errors or demographic bias in the predicted labels can directly affect the learned semantic boundaries <ref:2608.25862#pg4>. Future work will investigate whether learning semantic boundaries at intermediate denoising timesteps could yield better transferability or achieve a more favorable balance between demographic fairness and image quality <ref:2608.25862#pg4>. Another promising direction is to extend the timestep-decoupled semantic guidance beyond demographic attributes to more subtle semantic characteristics <ref:2608.25862#pg4>.

--- Page 1 ---

LEARNING LATE, GUIDING EARLY: TIMESTEP-DECOUPLED SEMANTIC GUIDANCE FOR FAIR FACE GENERATION<ref:2608.25862#pg2> Subir Kumar Parida∗ Rajbabu Velmurugan† Ketan Kotwal† R.S. Sengar∗ Swati Hiremath∗ Bhabha Atomic Research Centre (B.A.R.C.), Mumbai, India †Department of Electrical Engineering, Indian Institute of Technology Bombay, Mumbai, India August 28, 2026<ref:2608.25862#pg2>

ABSTRACT Demographic imbalance in synthetic face generation can propagate to downstream face recognition systems, making fairness an important consideration when diffusion models are used for data generation Existing fairness-aware generation approaches often require model retraining, architectural modifications, or repeated guidance throughout the reverse diffusion process In this work, we introduce Semantic Boundary Predictor (SBP), an inference-time framework that performs demographic guidance through a one-shot intervention during reverse denoising Our approach is motivated by the observation that latent representations at different diffusion timesteps play distinct semantic roles: late-stage latents provide stronger demographic separability, whereas early-stage latents offer greater flexibility for semantic intervention SBP leverages this timestep decoupling by learning linear semantic boundaries from late-stage latent representations while applying them only once at the initial noisy latent, allowing the remainder of the reverse denoising process to proceed unchanged. The method requires neither retraining nor fine-tuning of the underlying Latent Diffusion Model and operates without external balanced datasets. Experiments on CelebA-HQ demonstrate substantial improvements in demographic fairness, reducing fairness disparity by 98% for gender, 95% for binary race, and 15% for four-class race, while maintaining perceptual image quality across demographic groups. Owing to its one-shot inference strategy and model-agnostic design, SBP introduces only a small computational overhead and can be readily integrated with existing pre-trained latent diffusion models. Keywords Latent Diffusion Model · Demographic Fairness · Synthetic Data · Face Recognition · Generative AI · Inference-Time Guidance arXiv:2608.25862v2 [cs.CV] 27 Aug 2026

--- Page 1 ---

A PREPRINT - AUGUST 28, 2026 A PREPRINT - AUGUST 28,

How it works

The framework operates by decoupling semantic boundary learning from semantic guidance across diffusion timesteps SBP learns semantic boundaries from late-stage latent representations and applies them during the early denoising stage, where guidance is applied only once before the remainder of the reverse diffusion process proceeds unchanged.

1. textbf Semantic Boundary Predictor (SBP) Implementation for One-Shot Guidance. The SBP framework "learn[s] semantic boundaries from discriminative late-stage latent representations and applies them during early denoising stage, where guidance is applied only once before the remainder of the reverse diffusion process proceeds unchanged. This allows for inference-time demographic guidance without retraining or fine-tuning the diffusion model."

2. textbf Timestep Decoupling for Semantic Manipulation. The system exploits the observation that latent representations at different timesteps play distinct semantic roles: late-stage latents provide stronger demographic separability, whereas early-stage latents offer greater flexibility for semantic intervention. This decoupling enables effective inference-time demographic guidance without modifying or retraining the pre-trained diffusion model.

3. textbf Efficiency Gains via Latent Subspace Projection. The use of a compact-subspace variant of SBP that performs semantic boundary learning and guidance in a PCA subspace reduces computational cost by projecting features onto leading principal components, ensuring the method is more efficient without compromising its effectiveness.

4. textbf Multi-Attribute Generation Control. The framework allows for simultaneous control over multiple demographics by learning a separate boundary for each attribute, as demonstrated by the ability to perform simultaneous control of multiple attributes without retraining or modifying the underlying diffusion model.

5. textbf Robustness Across Demographic Attributes. The method is shown to generalize across various demographic challenges, achieving consistent improvement across gender, binary race (Black/White), four-class race (White, Black, Asian, Indian), and age, demonstrating that the learned semantic boundaries remain informative even as the demographic distribution becomes more complex.

Demographic imbalance in synthetic face generation can propagate to downstream face recognition systems, making fairness an important consideration when diffusion models are used for data generation. Existing fairness-aware generation approaches often require model retraining, architectural modifications, or repeated guidance throughout the reverse diffusion process. In this work, we introduce Semantic Boundary Predictor (SBP), an inference-time framework that performs demographic guidance through a one-shot intervention during reverse denoising. Our approach is motivated by the observation that latent representations at different diffusion timesteps play distinct semantic roles: late-stage latents provide stronger demographic separability, whereas early-stage latents offer greater flexibility for semantic intervention. SBP leverages this timestep decoupling by learning linear semantic boundaries from late-stage latent representations while applying them only once at the initial noisy latent, allowing the remainder of the reverse denoising process to proceed unchanged. The method requires neither retraining nor fine-tuning of the underlying Latent Diffusion Model and operates without external balanced datasets. Experiments on CelebA-HQ demonstrate substantial improvements in demographic fairness, reducing fairness disparity by 98% for gender, 95% for binary race, and 15% for four-class race, while maintaining perceptual image quality across demographic groups. Owing to its one-shot inference strategy and model-agnostic design, SBP introduces only a small computational overhead and can be readily integrated with existing pre-trained latent diffusion models.