eDMD: Endpoint Distribution Matching for Few-Step Diffusion Distillation
summary
The gist
Distribution matching distillation (DMD) has emerged as a leading paradigm for distilling diffusion models into few-step students…
In short
Endpoint Distribution Matching Distillation (eDMD) improves distilling diffusion models by training a single fake model only on the student's final outputs. This eliminates a residual error found in multi-step methods where intermediate predictions are used, ensuring distribution matching accurately reflects the final output. The method also enhances teacher supervision using precise trajectories and calibrated query density.
Key concepts
- Distribution Matching Distillation (DMD)
- A technique used to distill a large teacher diffusion model into a smaller student model. It works by training an auxiliary fake score model to track the student's distribution. The goal is for the student's output distribution to match the teacher's distribution, guiding the student toward high-quality generation.
- Endpoint Reference Residual
- In multi-step DMD, training a fake model on intermediate predictions creates an unwanted reference residual (R). This residual persists even when the final output matches the teacher's. eDMD resolves this by aligning the fake model's reference with only the endpoint distribution, causing this specific residual to vanish.
- Precise Teacher (PT)
- This mechanism improves teacher supervision by replacing single-step predictions with short multi-step trajectories focused on the final noise interval. This provides a more refined and accurate target for the student model during training, leading to better generation quality.
- Calibrated Query Density
- This technique concentrates intermediate-step queries onto noise levels where teacher predictions are most reliable. By focusing on these high-confidence supervision levels, eDMD strengthens the guidance signal for the student model, further improving distillation performance.
Terminology used across episodes
This episode discusses
- eDMD: Endpoint Distribution Matching for Few-Step Diffusion Distillation · Paper Radio
- Classifier-Free Guidance is a Predictor-Corrector
- Classifier-Free Diffusion Guidance
- ELLA: Equip Diffusion Models with LLM for Enhanced Semantic Alignment
- Reinforcing Few-step Generators via Reward-Tilted Distribution Matching
- 1.x-Distill: Breaking the Diversity, Quality, and Efficiency Barrier in Distribution Matching Distillation
- SDXL-Lightning: Progressive Adversarial Diffusion Distillation
- Continuous-Time Distribution Matching for Few-Step Diffusion Distillation
- HPSv3++: Scaling Reward Models Across the Full Spectrum of Diffusion Model Capabilities
- CrossDistill: Balancing Quality and Diversity via Trajectory-Level Hybrid Few-Step Distillation · Paper Radio
- DPM-Solver++: Fast Solver for Guided Sampling of Diffusion Probabilistic Models
- Latent Consistency Models: Synthesizing High-Resolution Images with Few-Step Inference
- Align Your Flow: Scaling Continuous-Time Flow Map Distillation
- Wan: Open and Advanced Large-Scale Video Generative Models
- Uni-Instruct: One-step Diffusion Model through Unified Diffusion Divergence Instruction
- Diversity-Preserved Distribution Matching Distillation for Fast Visual Synthesis
- Human Preference Score v2: A Solid Benchmark for Evaluating Human Preferences of Text-to-Image Synthesis
- SGMD: Score Gradient Matching Distillation for Few-Step Video Diffusion Distillation
- One-step Diffusion Models with f-Divergence Distribution Matching
- Few-Step Diffusion via Score identity Distillation
The paper
eDMD: Endpoint Distribution Matching for Few-Step Diffusion Distillation · Read on arXiv
Tong Zhao, Xianfang Zeng, Yucheng Han, Ruoyu Wang, Mingkun Lei, Gang Yu, Chi Zhang
Zhejiang University
Distribution matching distillation (DMD) has emerged as a leading paradigm for distilling diffusion models into few-step students, where an auxiliary fake score model tracks the student's distribution and the discrepancy between the teacher and fake predictions drives the student toward the teacher. In multi-step distillation, although only the student's final output is delivered, existing methods train the fake model on intermediate states or across all denoising steps. Consequently, the student's update signal becomes entangled with intermediate distributions, which can distort the matching direction and remain nonzero even after the student's final outputs already match the teacher distribution. To resolve this mismatch, we propose Endpoint Distribution Matching Distillation (eDMD), which trains a single shared fake model exclusively on the student's final outputs and uses it to correct every denoising step. Our theoretical analysis proves that this endpoint reference guarantees that the distribution matching term vanishes if and only if the student's final output distribution matches the teacher's, regardless of intermediate distributions. Beyond the fake reference, we further strengthen teacher supervision through a precise teacher (PT) that replaces single-step guided predictions with short multi-step trajectories on the final noise interval, paired with a calibrated query density for intermediate steps. Experiments on SD3-Medium, FLUX.2-klein (4B and 9B), and Qwen-Image-2.1 demonstrate that eDMD consistently outperforms existing distribution matching methods for 4-step generation.
Transcript
Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.
Tom: I'm Tom.
Jane: And I'm Jane, here with Lu, Meng and our in-house large language model, Lalam.
Tom: We're looking at the paper: "eDMD: Endpoint Distribution Matching for Few-Step Diffusion Distillation".
Jane: Here's the gist of the paper. Distribution matching distillation (DMD) has emerged as a leading paradigm for distilling diffusion models into few-step students, where an auxiliary fake score model tracks the student’s distribution and the discrepancy between the teacher and fake predictions drives the student toward the teacher.
Tom: The authors propose Endpoint Distribution Matching Distillation (eDMD), which trains a single shared fake model exclusively on the student’s final outputs and uses it to correct every denoising step.
Jane: This approach resolves a mismatch in existing multi-step DMD methods where training the fake model on intermediate predictions introduces an unwanted reference residual that can distort distribution matching, which vanishes if and only if the fake model learns the endpoint distribution.
Tom: Pruning redundant reasoning tokens The core theoretical finding is that existing multi-step DMD methods introduce a nonzero residual when training the fake model on intermediate predictions, which fundamentally changes the intended distribution matching behavior.
Paper summary: Tom: So, wrapping up "eDMD: Endpoint Distribution Matching for Few-Step Diffusion Distillation," we're talking about a method that trains a single fake model only on the final outputs to fix the distortion issue in multi-step distillation.
Jane: The authors are Tong Zhao, Xianfang Zeng, Yucheng Han, Ruoyu Wang, Mingkun Lei, and Gang Yu >
Lu: The core finding is that training the fake model on intermediate predictions introduces a reference residual that remains non-zero unless the fake model learns the endpoint distribution >
Meng: So they solve this by proposing eDMD which trains only on final outputs and uses that to correct every denoising step, while also using a precise teacher and calibrated query density for supervision >
Lalam: It’s important because it shows that distribution matching can be made robust to intermediate distributions if you correctly align the fake model’s reference with the endpoint distribution >
Tom: For someone listening who doesn't read papers, think of it this way: we used to train a guide model on every single tiny move a student made during a long process, but eDMD says that only caring about the final destination is what actually guides the student effectively >
Jane: That’s right. The paper suggests that for diffusion distillation into few-step students, focusing on the endpoint distribution is what makes those few steps work reliably >
Lu: It implies we can simplify our training signals significantly by ignoring intermediate states if we use a model trained purely on the final result >
Meng: If this holds up when we move these models to real traffic and massive scale, it could make distillation much more feasible than what we've seen before >
Conclusion: Tom: So, eDMD is basically taking that whole distribution matching idea and making it work for distilling diffusion models into just a few steps by focusing only on the final output, not all the intermediate steps.
Jane: That's right, Tom. The authors are Tong Zhao and his team, and they’re trying to fix a problem where if you train on every single tiny step during distillation, it messes up how well the student model learns from the teacher.
Lu: They found that training on those intermediate predictions creates this reference residual that stays around even when the final outputs match up perfectly, unless you specifically force the fake model to learn what happens at the very end.
Meng: From an engineering standpoint, that’s a headache because if that reference residual isn't gone, your distribution matching term just gets distorted by those middle steps instead of just being about getting to the right answer at the end.
Lalam: eDMD solves this by training a single fake model only on the final generated outputs and using that to correct every denoising step, which kills that residual issue entirely.
Tom: And they also added two things for better supervision: a precise teacher and a calibrated query density, which helps make sure the supervision signal is actually reliable across all those steps.
Jane: So what does this mean for us? It means we can distill these complex diffusion models into much shorter student models without worrying as much about getting the perfect intermediate path right.
Lu: The paper shows that focusing on that endpoint distribution makes the distribution matching term vanish if and only if the final outputs match, regardless of what happened in between.
Tom: It’s a solid result because they show it consistently outperforms other distillation methods across four different model backbones, hitting some of the highest scores we've seen for this specific task.
Meng: If this technique can be deployed reliably on real traffic at massive scale, it cuts down the computational cost of training these smaller models significantly.
Jane: Right, so they’re moving the focus from tracking every tiny move to just ensuring the final result is what we want, which simplifies things a lot for deployment.
More episodes
- 2603.29765-Training-Free Dynamic Upcycling of Expert Language Models
- 2605.00778-Observable Performance Does Not Fully Reflect Adaptive System Organization: A Multi-Level Analysis of Gait Dynamics Under Occlusal Constraint
- 2605.06906-TraXion: Rethinking Pre-training Frameworks for Mobility and Beyond
- 2605.09165-Sparse Layers are Critical to Scaling Looped Language Models
- 2605.09439-Inverse Design for Conditional Distribution Matching
- 2605.10313-Signature Approach for Contextual Bandits with Nonlinear and Path-dependent Rewards: Regret Bound and Statistical Inference
- 2605.10878-Minimum Neural Weight Norm = Kolmogorov Complexity
- 2605.11617-MIST: Reliable Streaming Decision Trees for Online Class-Incremental Learning via McDiarmid Bound
- 2605.15085-From Data to Action: Accelerating Refinery Optimization with AI
- 2605.23500-GRTO: Bootstrapped Group Relative Tool Optimization for Referring Segmentation