eDMD: Endpoint Distribution Matching for Few-Step Diffusion Distillation
Listen
Radio episode about this paper
Transcript
Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.
Tom: I'm Tom.
Jane: And I'm Jane, here with Lu, Meng and our in-house large language model, Lalam.
Tom: We're looking at the paper: "eDMD: Endpoint Distribution Matching for Few-Step Diffusion Distillation".
Jane: Here's the gist of the paper. Distribution matching distillation (DMD) has emerged as a leading paradigm for distilling diffusion models into few-step students, where an auxiliary fake score model tracks the student’s distribution and the discrepancy between the teacher and fake predictions drives the student toward the teacher.
Tom: The authors propose Endpoint Distribution Matching Distillation (eDMD), which trains a single shared fake model exclusively on the student’s final outputs and uses it to correct every denoising step.
Jane: This approach resolves a mismatch in existing multi-step DMD methods where training the fake model on intermediate predictions introduces an unwanted reference residual that can distort distribution matching, which vanishes if and only if the fake model learns the endpoint distribution.
Tom: Pruning redundant reasoning tokens The core theoretical finding is that existing multi-step DMD methods introduce a nonzero residual when training the fake model on intermediate predictions, which fundamentally changes the intended distribution matching behavior.
Paper summary: Tom: So, wrapping up "eDMD: Endpoint Distribution Matching for Few-Step Diffusion Distillation," we're talking about a method that trains a single fake model only on the final outputs to fix the distortion issue in multi-step distillation.
Jane: The authors are Tong Zhao, Xianfang Zeng, Yucheng Han, Ruoyu Wang, Mingkun Lei, and Gang Yu >
Lu: The core finding is that training the fake model on intermediate predictions introduces a reference residual that remains non-zero unless the fake model learns the endpoint distribution >
Meng: So they solve this by proposing eDMD which trains only on final outputs and uses that to correct every denoising step, while also using a precise teacher and calibrated query density for supervision >
Lalam: It’s important because it shows that distribution matching can be made robust to intermediate distributions if you correctly align the fake model’s reference with the endpoint distribution >
Tom: For someone listening who doesn't read papers, think of it this way: we used to train a guide model on every single tiny move a student made during a long process, but eDMD says that only caring about the final destination is what actually guides the student effectively >
Jane: That’s right. The paper suggests that for diffusion distillation into few-step students, focusing on the endpoint distribution is what makes those few steps work reliably >
Lu: It implies we can simplify our training signals significantly by ignoring intermediate states if we use a model trained purely on the final result >
Meng: If this holds up when we move these models to real traffic and massive scale, it could make distillation much more feasible than what we've seen before >
Conclusion: Tom: So, eDMD is basically taking that whole distribution matching idea and making it work for distilling diffusion models into just a few steps by focusing only on the final output, not all the intermediate steps.
Jane: That's right, Tom. The authors are Tong Zhao and his team, and they’re trying to fix a problem where if you train on every single tiny step during distillation, it messes up how well the student model learns from the teacher.
Lu: They found that training on those intermediate predictions creates this reference residual that stays around even when the final outputs match up perfectly, unless you specifically force the fake model to learn what happens at the very end.
Meng: From an engineering standpoint, that’s a headache because if that reference residual isn't gone, your distribution matching term just gets distorted by those middle steps instead of just being about getting to the right answer at the end.
Lalam: eDMD solves this by training a single fake model only on the final generated outputs and using that to correct every denoising step, which kills that residual issue entirely.
Tom: And they also added two things for better supervision: a precise teacher and a calibrated query density, which helps make sure the supervision signal is actually reliable across all those steps.
Jane: So what does this mean for us? It means we can distill these complex diffusion models into much shorter student models without worrying as much about getting the perfect intermediate path right.
Lu: The paper shows that focusing on that endpoint distribution makes the distribution matching term vanish if and only if the final outputs match, regardless of what happened in between.
Tom: It’s a solid result because they show it consistently outperforms other distillation methods across four different model backbones, hitting some of the highest scores we've seen for this specific task.
Meng: If this technique can be deployed reliably on real traffic at massive scale, it cuts down the computational cost of training these smaller models significantly.
Jane: Right, so they’re moving the focus from tracking every tiny move to just ensuring the final result is what we want, which simplifies things a lot for deployment.
Tong Zhao, Xianfang Zeng, Yucheng Han, Ruoyu Wang, Mingkun Lei, Gang Yu, Chi Zhang
Zhejiang University
cs.CV
Submitted: 2026-11-12
Updated: 2026-11-12
Comments: Code Link: see https://github.com/Westlake-AGI-Lab/eDMD
Code: https://github.com/Westlake-AGI-Lab/eDMD
License: http://creativecommons.org/licenses/by/4.0/
Importance score: 97/100
The gist: Distribution matching distillation (DMD) has emerged as a leading paradigm for distilling diffusion models into few-step students…
Key concepts
- Distribution Matching Distillation (DMD)
- A technique used to distill a large teacher diffusion model into a smaller student model. It works by training an auxiliary fake score model to track the student's distribution. The goal is for the student's output distribution to match the teacher's distribution, guiding the student toward high-quality generation.
- Endpoint Reference Residual
- In multi-step DMD, training a fake model on intermediate predictions creates an unwanted reference residual (R). This residual persists even when the final output matches the teacher's. eDMD resolves this by aligning the fake model's reference with only the endpoint distribution, causing this specific residual to vanish.
- Precise Teacher (PT)
- This mechanism improves teacher supervision by replacing single-step predictions with short multi-step trajectories focused on the final noise interval. This provides a more refined and accurate target for the student model during training, leading to better generation quality.
- Calibrated Query Density
- This technique concentrates intermediate-step queries onto noise levels where teacher predictions are most reliable. By focusing on these high-confidence supervision levels, eDMD strengthens the guidance signal for the student model, further improving distillation performance.
Terminology
Summary
Distribution matching distillation (DMD) has emerged as a leading paradigm for distilling diffusion models into few-step students, where an auxiliary fake score model tracks the student’s distribution and the discrepancy between the teacher and fake predictions drives the student toward the teacher. p. 0 Abstract
Distribution matching distillation (DMD) has emerged as a
leading paradigm for distilling diffusion models into fewstep students, where an auxiliary fake score model tracks
the student’s distribution and the discrepancy between the
teacher and fake predictions drives the student toward the
teacher. In multi-step distillation, although only the stu-
†Corresponding author: chizhang@westlake. Consequently, the student’s update signal becomes
entangled with intermediate distributions, which can distort the matching direction and remain nonzero even after
the student’s final outputs already match the teacher distribution. To resolve this mismatch, we propose Endpoint
Distribution Matching Distillation (eDMD), which trains a
single shared fake model exclusively on the student’s final
outputs and uses it to correct every denoising step. Extensive experiments
demonstrate that eDMD consistently outperforms existing
distribution matching baselines, achieving the highest peak
HPSv3++ [26] scores across all four backbones. Our main
contributions are summarized as follows:
• We identify and theoretically characterize an objective
mismatch in multi-step DMD: training the fake model
on intermediate predictions introduces a reference residual that distorts distribution matching and vanishes if and
only if the fake model learns the endpoint distribution.Where the paper says it
Where the paper says it
Where the paper says it
Pruning redundant reasoning tokens
The core theoretical finding is that existing multi-step DMD methods introduce a nonzero residual when training the fake model on intermediate predictions, which fundamentally changes the intended distribution matching behavior. This mismatch is characterized by a reference residual R which remains nonzero even when the student’s final output distribution matches the teacher’s, and this residual vanishes if and only if the fake model represents the endpoint distribution. p. 1 Even when
the student’s final output distribution already matches the
teacher’s, this residual may remain nonzero and continue
to push the student away from the desired distribution. We
further prove that the reference residual vanishes if and
only if the fake model represents the endpoint distribution,
regardless of the student’s intermediate distributions
(Prop. Even when
the student’s final output distribution already matches the
teacher’s, this residual may remain nonzero and continue
to push the student away from the desired distribution. We
further prove that the reference residual vanishes if and
only if the fake model represents the endpoint distribution,
regardless of the student’s intermediate distributions
(Prop. oretical analysis proves that this endpoint reference guarantees that the distribution matching term vanishes if and
only if the student’s final output distribution matches the
teacher’s, regardless of intermediate distributions. Beyond
the fake reference, we further strengthen teacher supervision through a precise teacher (PT) that replaces singlestep guided predictions with short multi-step trajectories on
the final noise interval, paired with a calibrated query density for intermediate steps.Where the paper says it
Where the paper says it
Where the paper says it
How the training signal is built
eDMD trains a single shared fake model exclusively on the student’s final generated outputs to eliminate this reference residual. The training objective involves minimizing a surrogate loss where the correction C is defined as C = A − F at every step, where A is the teacher's prediction and F is the fake model's prediction. To strengthen supervision, eDMD introduces two key components: a precise teacher (PT) that replaces single-step predictions with short multi-step trajectories on the final noise interval, and a calibrated query density to concentrate intermediate-step queries on reliable supervision levels. p. 1 oretical analysis proves that this endpoint reference guarantees that the distribution matching term vanishes if and
only if the student’s final output distribution matches the
teacher’s, regardless of intermediate distributions. Beyond
the fake reference, we further strengthen teacher supervision through a precise teacher (PT) that replaces singlestep guided predictions with short multi-step trajectories on
the final noise interval, paired with a calibrated query density for intermediate steps.Where the paper says it
Where the gains come from
The primary gains stem from aligning the fake model’s reference with the endpoint distribution, which eliminates the reference residual and ensures that distribution matching is determined by the discrepancy between teacher and student final outputs independently of intermediate sampling trajectory. Furthermore, eDMD improves teacher supervision through two mechanisms: a precise teacher (PT) that refines low-noise targets through multi-step denoising, and a calibrated query density that favors noise levels where teacher predictions are more reliable. p. 8 We further improve teacher supervision in
two ways: a precise teacher (PT) refines low-noise targets
through multi-step denoising, and a calibrated query
density favors noise levels where teacher predictions are
more reliable. Experiments on four diffusion and flow
matching backbones show that eDMD improves 4-step generation quality over existing distribution matching methods. 1 [38]), distilling each into a 4-step generator without using real images. Extensive experiments
demonstrate that eDMD consistently outperforms existing
distribution matching baselines, achieving the highest peak
HPSv3++ [26] scores across all four backbones.Where the paper says it
Where the paper says it
The authors identify and theoretically characterize an objective mismatch in multi-step DMD where training the fake model on intermediate predictions introduces a reference residual that distorts distribution matching and vanishes if and only if the fake model learns the endpoint distribution. This is further detailed by analyzing three types of references: mixture reference, stage reference, and endpoint reference. p. 4 For the
mismatched student, all three references contain the desired
update, but the mixture and stage references add a residual
that deflects it. For the matched student, the desired update is zero and the endpoint reference yields no correction,
whereas the mixture and stage references still push the student away from the teacher. The students whose correction vanishes
across all queries and steps are
endpoint: {q = p},
mixture: {
P
k
πkqk = p},
stage: {q = p, rk = pσk+1 ∀k < K − 1}. (6)
Only the endpoint reference stops exactly at q = p.Where the paper says it
Where the paper says it
The authors propose eDMD, which trains a shared fake model exclusively on the student’s final generated outputs and uses it to provide correction signals for every denoising step. The loss function used for training the fake model is defined as LF = E h λ(τ) F((1 − τ)x + τϵ, τ − x / 2i, (7). p. 2 Right: compared with DMD2 and TDM, eDMD trains the fake model on final outputs only (top),
draws the queries of intermediate steps from a calibrated density (middle; ticks: the level each step denoises to), and answers low-noise
queries with a multi-step teacher instead of a single prediction (bottom; percentages: answers on the tree). model should represent the distribution of the samples that
distillation ultimately aims to optimize, rather than the intermediate predictions used to generate them. Right: compared with DMD2 and TDM, eDMD trains the fake model on final outputs only (top),
draws the queries of intermediate steps from a calibrated density (middle; ticks: the level each step denoises to), and answers low-noise
queries with a multi-step teacher instead of a single prediction (bottom; percentages: answers on the tree). model should represent the distribution of the samples that
distillation ultimately aims to optimize, rather than the intermediate predictions used to generate them.Where the paper says it
Where the paper says it
The authors demonstrate that eDMD improves 4-step generation quality over existing distribution matching methods, achieving the highest peak HPSv3++ scores across all four backbones. The ablation studies confirm that removing the endpoint reference causes the largest overall degradation among component ablations on both SD3-Medium and FLUX. p. 8 Endpoint reference. Removing E causes the largest overall degradation among the component ablations on both
SD3-Medium and FLUX. Experiments on SD3-Medium,
FLUX. 2-klein (4B and 9B), and Qwen-Image-2. We further improve teacher supervision in
two ways: a precise teacher (PT) refines low-noise targets
through multi-step denoising, and a calibrated query
density favors noise levels where teacher predictions are
more reliable. Experiments on four diffusion and flow
matching backbones show that eDMD improves 4-step generation quality over existing distribution matching methods. 1 [38]), distilling each into a 4-step generator without using real images. Extensive experiments
demonstrate that eDMD consistently outperforms existing
distribution matching baselines, achieving the highest peak
HPSv3++ [26] scores across all four backbones.Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
The authors show that training the fake model on intermediate predictions or states can distort distribution matching in multi-step DMD. The resulting reference residual can persist even after the student’s final output distribution matches the teacher’s. p. 8 Conclusion
We showed that training the fake model on intermediate
predictions or states can distort distribution matching in
multi-step DMD. The resulting reference residual can
persist even after the student’s final output distribution
matches the teacher’s. eDMD resolves this mismatch
with an endpoint reference, training a shared fake model
exclusively on final outputs and using it to supervise
every denoising step. This keeps distribution matching
focused on the samples delivered at inference and gives
intermediate predictions the freedom to adapt to few-step
generation.Where the paper says it
Where the paper says it
The authors propose eDMD, which trains a single shared fake model exclusively on the student’s final generated outputs and uses it to provide correction signals for every denoising step. The loss function used for training the fake model is defined as LF = E h λ(τ) F((1 − τ)x + τϵ, τ − x / 2i, (7). The resulting expected parameter update follows a negative gradient pushing queries toward higher log(pτ /qτ).
The authors identify and theoretically characterize an objective mismatch in multi-step DMD where training the fake model on intermediate predictions introduces a reference residual that distorts distribution matching and vanishes if and only if the fake model learns the endpoint distribution. This mismatch is characterized by a reference residual R which remains nonzero even when the student’s final output distribution matches the teacher’s, and this residual vanishes if and only if the fake model represents the endpoint distribution.
Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.
Improvements for AI systems
-
Endpoint Distribution Matching Distillation (eDMD) eliminates reference residual by training a single shared fake model exclusively on student final outputs and using it to correct every denoising step, ensuring
the distribution matching signal is determined by the discrepancy between the teacher and the student’s final outputs, independently of the intermediate sampling trajectory.
-
Teacher supervision is strengthened via a precise teacher (PT) that replaces singlestep guided predictions with short multi-step trajectories on the final noise interval and a calibrated query density to concentrate intermediate-step queries on noise levels providing more reliable supervision.
-
The system can preserve fine details in complex structures like fur, hair, and grass better than existing methods, achieving
higher human-preference scores
andfewer structural artifacts
across various backbones.
Abstract
Distribution matching distillation (DMD) has emerged as a leading paradigm for distilling diffusion models into few-step students, where an auxiliary fake score model tracks the student's distribution and the discrepancy between the teacher and fake predictions drives the student toward the teacher. In multi-step distillation, although only the student's final output is delivered, existing methods train the fake model on intermediate states or across all denoising steps. Consequently, the student's update signal becomes entangled with intermediate distributions, which can distort the matching direction and remain nonzero even after the student's final outputs already match the teacher distribution. To resolve this mismatch, we propose Endpoint Distribution Matching Distillation (eDMD), which trains a single shared fake model exclusively on the student's final outputs and uses it to correct every denoising step. Our theoretical analysis proves that this endpoint reference guarantees that the distribution matching term vanishes if and only if the student's final output distribution matches the teacher's, regardless of intermediate distributions. Beyond the fake reference, we further strengthen teacher supervision through a precise teacher (PT) that replaces single-step guided predictions with short multi-step trajectories on the final noise interval, paired with a calibrated query density for intermediate steps. Experiments on SD3-Medium, FLUX.2-klein (4B and 9B), and Qwen-Image-2.1 demonstrate that eDMD consistently outperforms existing distribution matching methods for 4-step generation.
Sources
- Classifier-Free Guidance is a Predictor-Corrector
- Classifier-Free Diffusion Guidance
- ELLA: Equip Diffusion Models with LLM for Enhanced Semantic Alignment
- Reinforcing Few-step Generators via Reward-Tilted Distribution Matching
- 1.x-Distill: Breaking the Diversity, Quality, and Efficiency Barrier in Distribution Matching Distillation
- SDXL-Lightning: Progressive Adversarial Diffusion Distillation
- Continuous-Time Distribution Matching for Few-Step Diffusion Distillation
- HPSv3++: Scaling Reward Models Across the Full Spectrum of Diffusion Model Capabilities
- CrossDistill: Balancing Quality and Diversity via Trajectory-Level Hybrid Few-Step Distillation
- DPM-Solver++: Fast Solver for Guided Sampling of Diffusion Probabilistic Models
- Latent Consistency Models: Synthesizing High-Resolution Images with Few-Step Inference
- Align Your Flow: Scaling Continuous-Time Flow Map Distillation
- Wan: Open and Advanced Large-Scale Video Generative Models
- Uni-Instruct: One-step Diffusion Model through Unified Diffusion Divergence Instruction
- Diversity-Preserved Distribution Matching Distillation for Fast Visual Synthesis
- Human Preference Score v2: A Solid Benchmark for Evaluating Human Preferences of Text-to-Image Synthesis
- SGMD: Score Gradient Matching Distillation for Few-Step Video Diffusion Distillation
- One-step Diffusion Models with $f$-Divergence Distribution Matching
- Few-Step Diffusion via Score identity Distillation
Related papers
- TEA: Text Encoder Alignment for Robust Concept Erasure in Text-to-Image Models
- Do Egocentric Video-Language Models Capture Both Hand- and Object-Centric Cues?
- Aura: Consistent Multi-Subject Video Generation via VLM-Grounded Semantic Alignment
- PVRF: All-in-one Adverse Weather Removal via Prior-modulated and Velocity-constrained Rectified Flow
- DLWM: Diverse Latent World Models for Efficient Multimodal Reasoning
- Out of Sight, Still in Mind: Token Compression for Omni-LLMs