Steering Training Trajectories in Diffusion Representation Learning
cs.CV, cs.AI
Submitted: 2026-07-06
Updated: 2026-11-06
Comments: Proceedings of the Fortieth Annual Conference on Neural Information Processing Systems, Sydney, Australia
Code: https://github.com/crowsonkb/k-diffusion
License: http://creativecommons.org/licenses/by/4.0/
The gist: Across training runs of a diffusion autoencoder, we observe similar image reconstruction quality, but substantially different levels of disentanglement in the learned representations.
Terminology
Abstract
Across training runs of a diffusion autoencoder, we observe similar image reconstruction quality, but substantially different levels of disentanglement in the learned representations. In this work, we study how training dynamics contribute to this behaviour. By analysing training trajectories in a controlled setting - curves of reconstruction quality against disentanglement during training - we identify two recurring trajectory patterns, which we call learning regimes. These regimes differ in how reconstruction and disentanglement evolve, despite similar reconstruction at the end of training. In the reconstruction regime, models improve image fidelity before disentanglement, while in the disentanglement regime, models improve both reconstruction and disentanglement more gradually. We hypothesise that modulating shortcut pathways in the diffusion U-Net and controlling early noise-level exposure can shape trajectories. To that end, we introduce SteeringDRL, combining gated residual U-Nets with a noise curriculum to steer models towards better disentanglement. Across attribute disentanglement benchmarks, SteeringDRL has competitive disentanglement, with reduced seed sensitivity in many settings. Our method extends to spatial disentanglement with object-centric learning, improving unsupervised segmentation on synthetic and real-world datasets.
Sources
- Interaction Asymmetry: A General Principle for Learning Composable Abstractions
- Understanding disentangling in $\beta$-VAE
- MONet: Unsupervised Scene Decomposition and Representation
- RoentGen: Vision-Language Foundation Model for Chest X-ray Generation
- On the Importance of Noise Scheduling for Diffusion Models
- Disentangled Representation Learning via Flow Matching
- Generating Long Sequences with Sparse Transformers
- GENESIS: Generative Scene Inference and Sampling with Object-Centric Latent Representations
- Unified Latents (UL): How to train your latents
- Prompt-to-Prompt Image Editing with Cross Attention Control
- Simpler Diffusion (SiD2): 1.5 FID on ImageNet512 with pixel-space diffusion
- Object-Centric Slot Diffusion
- Learning to Compose: Improving Object Centric Learning by Injecting Compositionality
- Disentangled Representation Learning via Modular Compositional Bias
- DiffusionSat: A Generative Foundation Model for Satellite Imagery
- Adaptive Non-uniform Timestep Sampling for Accelerating Diffusion Model Training
- FLUX.1 Kontext: Flow Matching for In-Context Image Generation and Editing in Latent Space
- Back to Basics: Let Denoising Generative Models Denoise
- Understanding Representation Dynamics of Diffusion Models via Low-Dimensional Modeling
- Flow Matching for Generative Modeling
Related papers
- TEA: Text Encoder Alignment for Robust Concept Erasure in Text-to-Image Models
- Do Egocentric Video-Language Models Capture Both Hand- and Object-Centric Cues?
- Aura: Consistent Multi-Subject Video Generation via VLM-Grounded Semantic Alignment
- PVRF: All-in-one Adverse Weather Removal via Prior-modulated and Velocity-constrained Rectified Flow
- DLWM: Diverse Latent World Models for Efficient Multimodal Reasoning
- Out of Sight, Still in Mind: Token Compression for Omni-LLMs