Stabilizing Consistency Training: A Flow Map Analysis and Self-Distillation
cs.LG, cs.CV
Submitted: 2026-01-30
Updated: 2026-11-01
Comments: 45 pages, 20 figures, 15 tables. Project page: https://snu-vgilab.github.io/improvedSelfDistillation/
Journal ref: Transactions on Machine Learning Research (10/2026)
License: http://creativecommons.org/licenses/by-nc-sa/4.0/
The gist: Consistency models have been proposed for fast generative modeling, achieving results competitive with diffusion and flow models.
Terminology
Abstract
Consistency models have been proposed for fast generative modeling, achieving results competitive with diffusion and flow models. However, these methods exhibit instability and limited reproducibility when training from scratch, motivating subsequent work to explain and stabilize these issues. While these efforts have provided valuable insights, the explanations remain fragmented, and the theoretical relationships remain unclear. In this work, we provide a theoretical examination of consistency models by analyzing them from a flow map-based perspective. This joint analysis clarifies how training and convergence behavior can give rise to degenerate solutions. Building on these insights, we revisit self-distillation as a practical remedy for certain forms of suboptimal convergence. To avoid excessive gradient norms during its optimization, we reformulate its objective empirically, yielding improved Self-Distillation (iSD). We further extend iSD with training-time classifier-free guidance, yielding iSD-T, which improves performance and reproducibility in the studied settings. We demonstrate that iSD extends beyond image generation to diffusion-based policy learning, without reliance on pretrained diffusion models for initialization.
Sources
Related papers
- FlexMoE: Learning One-for-All Multi-Budget Expert Pruning for Flexible MoE Deployment
- OptiMind: Improving Small Language Models for Optimization Modeling through Formulation Guidance and Multi-Turn Learning
- Staying in the Sweet Spot: Boost Reasoning Evolution via Responsive Rollout Difficulty Manipulation
- Examining the Impact of Data and Training Methods on Emergent and Subliminal Misalignment
- Rethinking Black-Box On-Policy Distillation: A Rubric-based Analysis
- (Mis)generalization of Helpful-Only Fine-tuning