Optimal Self-Distillation for Rectified Flow via Linear Probing
Saptarshi Roy, Debepsita Mukherjee, Pratik Patil
stat.ML, cs.LG
Submitted: 2026-07-16
Comments: 29 pages
License: http://creativecommons.org/licenses/by/4.0/
The gist: Modern generative models are increasingly trained using model-generated signals, creating both opportunities for self-improvement and risks of collapse.
Terminology
Abstract
Modern generative models are increasingly trained using model-generated signals, creating both opportunities for self-improvement and risks of collapse. We study optimal self-distillation (SD) for rectified flow (RF): given a suboptimal teacher velocity field, can a student trained on a mixture of true RF velocities and teacher velocities provably improve the teacher? For linear RF with ridge regularization on fixed interpolation pairs, we prove an exact affine path identity, derive the optimal mixing coefficient in closed form, and show strict improvement in integrated velocity risk whenever the teacher risk is nonstationary along the regularization path. The optimal coefficient obeys a sign rule: positive mixing corrects under-regularized teachers, while negative mixing corrects over-regularized teachers. We also give one-shot generalized cross-validation (GCV) and validation tuning procedure that avoids grid search over mixing weights and repeated refitting. Combining this theorem with RF Wasserstein convergence bounds, we show that optimal self-distillation improves the velocity estimation terms controlling continuous-time and finite-step generation error. Experiments with Gaussian models, Gaussian mixtures, and image data show that optimal self-distillation improves velocity risk, mode recovery, and finite-step generation relative to both the teacher and pure distillation.
Sources
- Building Normalizing Flows with Stochastic Interpolants
- Self-Consuming Generative Models Go MAD
- On the Convergence and Straightness of Rectified Flow
- Nepotistically Trained Generative-AI Models Collapse
- Optimal Unconstrained Self-Distillation in Ridge Regression: Strict Improvements, Precise Asymptotics, and One-Shot Tuning
- Understanding Self-Distillation in the Presence of Label Noise
- Scaling Rectified Flow Transformers for High-Resolution Image Synthesis
- Towards Theoretical Understandings of Self-Consuming Generative Models
- Born Again Neural Networks
- Will Large-scale Generative Models Corrupt Future Datasets?
- Distilling the Knowledge in a Neural Network
- Knowledge Distillation in Wide Neural Networks: Risk Bound, Data Efficiency and Imperfect Teacher
- Pyramidal Flow Matching for Efficient Video Generative Modeling
- Dynamic Negative Guidance of Diffusion Models
- Voicebox: Text-Guided Multilingual Universal Speech Generation at Scale
- Flow Matching for Generative Modeling
- Flow Straight and Fast: Learning to Generate and Transfer Data with Rectified Flow
- A Comprehensive Survey on Knowledge Distillation of Diffusion Models
- Combining Generative Artificial Intelligence (AI) and the Internet: Heading towards Evolution or Degradation?
- Self-Distillation Amplifies Regularization in Hilbert Space
Related papers
- Behavior of prediction performance metrics with rare events
- Optimal Estimation of Generic Dynamics by Path-Dependent Neural Jump ODEs
- A Posterior-Dynamics Framework for Imaging Inverse Problems with Pretrained Diffusion Priors
- Scalable Posterior Uncertainty for Flexible Density-Based Clustering
- A Transport Geometry of Benign Overfitting
- Efficient quadratic entropy with distance sketches