eDMD: Endpoint Distribution Matching for Few-Step Diffusion Distillation

arXiv:2611.17194 · cs.CV · Submitted 2026-11-12 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: I'm Tom.

Jane: And I'm Jane, here with Lu, Meng and our in-house large language model, Lalam.

Tom: We're looking at the paper: "eDMD: Endpoint Distribution Matching for Few-Step Diffusion Distillation".

Jane: Here's the gist of the paper. Distribution matching distillation (DMD) has emerged as a leading paradigm for distilling diffusion models into few-step students, where an auxiliary fake score model tracks the student’s distribution and the discrepancy between the teacher and fake predictions drives the student toward the teacher.

Tom: The authors propose Endpoint Distribution Matching Distillation (eDMD), which trains a single shared fake model exclusively on the student’s final outputs and uses it to correct every denoising step.

Jane: This approach resolves a mismatch in existing multi-step DMD methods where training the fake model on intermediate predictions introduces an unwanted reference residual that can distort distribution matching, which vanishes if and only if the fake model learns the endpoint distribution.

Tom: Pruning redundant reasoning tokens The core theoretical finding is that existing multi-step DMD methods introduce a nonzero residual when training the fake model on intermediate predictions, which fundamentally changes the intended distribution matching behavior.

Paper summary: Tom: So, wrapping up "eDMD: Endpoint Distribution Matching for Few-Step Diffusion Distillation," we're talking about a method that trains a single fake model only on the final outputs to fix the distortion issue in multi-step distillation.

Jane: The authors are Tong Zhao, Xianfang Zeng, Yucheng Han, Ruoyu Wang, Mingkun Lei, and Gang Yu >

Lu: The core finding is that training the fake model on intermediate predictions introduces a reference residual that remains non-zero unless the fake model learns the endpoint distribution >

Meng: So they solve this by proposing eDMD which trains only on final outputs and uses that to correct every denoising step, while also using a precise teacher and calibrated query density for supervision >

Lalam: It’s important because it shows that distribution matching can be made robust to intermediate distributions if you correctly align the fake model’s reference with the endpoint distribution >

Tom: For someone listening who doesn't read papers, think of it this way: we used to train a guide model on every single tiny move a student made during a long process, but eDMD says that only caring about the final destination is what actually guides the student effectively >

Jane: That’s right. The paper suggests that for diffusion distillation into few-step students, focusing on the endpoint distribution is what makes those few steps work reliably >

Lu: It implies we can simplify our training signals significantly by ignoring intermediate states if we use a model trained purely on the final result >

Meng: If this holds up when we move these models to real traffic and massive scale, it could make distillation much more feasible than what we've seen before >

Conclusion: Tom: So, eDMD is basically taking that whole distribution matching idea and making it work for distilling diffusion models into just a few steps by focusing only on the final output, not all the intermediate steps.

Jane: That's right, Tom. The authors are Tong Zhao and his team, and they’re trying to fix a problem where if you train on every single tiny step during distillation, it messes up how well the student model learns from the teacher.

Lu: They found that training on those intermediate predictions creates this reference residual that stays around even when the final outputs match up perfectly, unless you specifically force the fake model to learn what happens at the very end.

Meng: From an engineering standpoint, that’s a headache because if that reference residual isn't gone, your distribution matching term just gets distorted by those middle steps instead of just being about getting to the right answer at the end.

Lalam: eDMD solves this by training a single fake model only on the final generated outputs and using that to correct every denoising step, which kills that residual issue entirely.

Tom: And they also added two things for better supervision: a precise teacher and a calibrated query density, which helps make sure the supervision signal is actually reliable across all those steps.

Jane: So what does this mean for us? It means we can distill these complex diffusion models into much shorter student models without worrying as much about getting the perfect intermediate path right.

Lu: The paper shows that focusing on that endpoint distribution makes the distribution matching term vanish if and only if the final outputs match, regardless of what happened in between.

Tom: It’s a solid result because they show it consistently outperforms other distillation methods across four different model backbones, hitting some of the highest scores we've seen for this specific task.

Meng: If this technique can be deployed reliably on real traffic at massive scale, it cuts down the computational cost of training these smaller models significantly.

Jane: Right, so they’re moving the focus from tracking every tiny move to just ensuring the final result is what we want, which simplifies things a lot for deployment.

Tong Zhao, Xianfang Zeng, Yucheng Han, Ruoyu Wang, Mingkun Lei, Gang Yu, Chi Zhang

Zhejiang University

cs.CV

Submitted: 2026-11-12

Updated: 2026-11-12

Comments: Code Link: see https://github.com/Westlake-AGI-Lab/eDMD

Code: https://github.com/Westlake-AGI-Lab/eDMD

License: http://creativecommons.org/licenses/by/4.0/

Importance score: 97/100

The gist: Distribution matching distillation (DMD) has emerged as a leading paradigm for distilling diffusion models into few-step students…

Key concepts

Distribution Matching Distillation (DMD)
A technique used to distill a large teacher diffusion model into a smaller student model. It works by training an auxiliary fake score model to track the student's distribution. The goal is for the student's output distribution to match the teacher's distribution, guiding the student toward high-quality generation.
Endpoint Reference Residual
In multi-step DMD, training a fake model on intermediate predictions creates an unwanted reference residual (R). This residual persists even when the final output matches the teacher's. eDMD resolves this by aligning the fake model's reference with only the endpoint distribution, causing this specific residual to vanish.
Precise Teacher (PT)
This mechanism improves teacher supervision by replacing single-step predictions with short multi-step trajectories focused on the final noise interval. This provides a more refined and accurate target for the student model during training, leading to better generation quality.
Calibrated Query Density
This technique concentrates intermediate-step queries onto noise levels where teacher predictions are most reliable. By focusing on these high-confidence supervision levels, eDMD strengthens the guidance signal for the student model, further improving distillation performance.

Terminology

Summary

Distribution matching distillation (DMD) has emerged as a leading paradigm for distilling diffusion models into few-step students, where an auxiliary fake score model tracks the student’s distribution and the discrepancy between the teacher and fake predictions drives the student toward the teacher. p. 0

Where the paper says it

Abstract Distribution matching distillation (DMD) has emerged as a leading paradigm for distilling diffusion models into fewstep students, where an auxiliary fake score model tracks the student’s distribution and the discrepancy between the teacher and fake predictions drives the student toward the teacher. In multi-step distillation, although only the stu- †Corresponding author: chizhang@westlake.

The authors propose Endpoint Distribution Matching Distillation (eDMD), which trains a single shared fake model exclusively on the student’s final outputs and uses it to correct every denoising step. p. 0
Where the paper says it

Consequently, the student’s update signal becomes entangled with intermediate distributions, which can distort the matching direction and remain nonzero even after the student’s final outputs already match the teacher distribution. To resolve this mismatch, we propose Endpoint Distribution Matching Distillation (eDMD), which trains a single shared fake model exclusively on the student’s final outputs and uses it to correct every denoising step.

This approach resolves a mismatch in existing multi-step DMD methods where training the fake model on intermediate predictions introduces an unwanted reference residual that can distort distribution matching, which vanishes if and only if the fake model learns the endpoint distribution. p. 2
Where the paper says it

Extensive experiments demonstrate that eDMD consistently outperforms existing distribution matching baselines, achieving the highest peak HPSv3++ [26] scores across all four backbones. Our main contributions are summarized as follows: • We identify and theoretically characterize an objective mismatch in multi-step DMD: training the fake model on intermediate predictions introduces a reference residual that distorts distribution matching and vanishes if and only if the fake model learns the endpoint distribution.

Pruning redundant reasoning tokens

The core theoretical finding is that existing multi-step DMD methods introduce a nonzero residual when training the fake model on intermediate predictions, which fundamentally changes the intended distribution matching behavior. This mismatch is characterized by a reference residual R which remains nonzero even when the student’s final output distribution matches the teacher’s, and this residual vanishes if and only if the fake model represents the endpoint distribution. p. 1

Where the paper says it

Even when the student’s final output distribution already matches the teacher’s, this residual may remain nonzero and continue to push the student away from the desired distribution. We further prove that the reference residual vanishes if and only if the fake model represents the endpoint distribution, regardless of the student’s intermediate distributions (Prop.

p. 1
Where the paper says it

Even when the student’s final output distribution already matches the teacher’s, this residual may remain nonzero and continue to push the student away from the desired distribution. We further prove that the reference residual vanishes if and only if the fake model represents the endpoint distribution, regardless of the student’s intermediate distributions (Prop.

The authors prove that this endpoint reference guarantees that the distribution matching term vanishes if and only if the student’s final output distribution matches the teacher’s, regardless of intermediate distributions. p. 1
Where the paper says it

oretical analysis proves that this endpoint reference guarantees that the distribution matching term vanishes if and only if the student’s final output distribution matches the teacher’s, regardless of intermediate distributions. Beyond the fake reference, we further strengthen teacher supervision through a precise teacher (PT) that replaces singlestep guided predictions with short multi-step trajectories on the final noise interval, paired with a calibrated query density for intermediate steps.

How the training signal is built

eDMD trains a single shared fake model exclusively on the student’s final generated outputs to eliminate this reference residual. The training objective involves minimizing a surrogate loss where the correction C is defined as C = A − F at every step, where A is the teacher's prediction and F is the fake model's prediction. To strengthen supervision, eDMD introduces two key components: a precise teacher (PT) that replaces single-step predictions with short multi-step trajectories on the final noise interval, and a calibrated query density to concentrate intermediate-step queries on reliable supervision levels. p. 1

Where the paper says it

oretical analysis proves that this endpoint reference guarantees that the distribution matching term vanishes if and only if the student’s final output distribution matches the teacher’s, regardless of intermediate distributions. Beyond the fake reference, we further strengthen teacher supervision through a precise teacher (PT) that replaces singlestep guided predictions with short multi-step trajectories on the final noise interval, paired with a calibrated query density for intermediate steps.

Where the gains come from

The primary gains stem from aligning the fake model’s reference with the endpoint distribution, which eliminates the reference residual and ensures that distribution matching is determined by the discrepancy between teacher and student final outputs independently of intermediate sampling trajectory. Furthermore, eDMD improves teacher supervision through two mechanisms: a precise teacher (PT) that refines low-noise targets through multi-step denoising, and a calibrated query density that favors noise levels where teacher predictions are more reliable. p. 8

Where the paper says it

We further improve teacher supervision in two ways: a precise teacher (PT) refines low-noise targets through multi-step denoising, and a calibrated query density favors noise levels where teacher predictions are more reliable. Experiments on four diffusion and flow matching backbones show that eDMD improves 4-step generation quality over existing distribution matching methods.

Experiments demonstrate that eDMD consistently outperforms existing distribution matching methods for 4-step generation across four representative backbones, achieving the highest peak HPSv3++ scores. p. 2
Where the paper says it

1 [38]), distilling each into a 4-step generator without using real images. Extensive experiments demonstrate that eDMD consistently outperforms existing distribution matching baselines, achieving the highest peak HPSv3++ [26] scores across all four backbones.

The authors identify and theoretically characterize an objective mismatch in multi-step DMD where training the fake model on intermediate predictions introduces a reference residual that distorts distribution matching and vanishes if and only if the fake model learns the endpoint distribution. This is further detailed by analyzing three types of references: mixture reference, stage reference, and endpoint reference. p. 4

Where the paper says it

For the mismatched student, all three references contain the desired update, but the mixture and stage references add a residual that deflects it. For the matched student, the desired update is zero and the endpoint reference yields no correction, whereas the mixture and stage references still push the student away from the teacher.

The analysis shows that only the endpoint reference stops exactly at q = p. p. 4
Where the paper says it

The students whose correction vanishes across all queries and steps are endpoint: {q = p}, mixture: { P k πkqk = p}, stage: {q = p, rk = pσk+1 ∀k < K − 1}. (6) Only the endpoint reference stops exactly at q = p.

The authors propose eDMD, which trains a shared fake model exclusively on the student’s final generated outputs and uses it to provide correction signals for every denoising step. The loss function used for training the fake model is defined as LF = E h λ(τ) F((1 − τ)x + τϵ, τ − x / 2i, (7). p. 2

Where the paper says it

Right: compared with DMD2 and TDM, eDMD trains the fake model on final outputs only (top), draws the queries of intermediate steps from a calibrated density (middle; ticks: the level each step denoises to), and answers low-noise queries with a multi-step teacher instead of a single prediction (bottom; percentages: answers on the tree). model should represent the distribution of the samples that distillation ultimately aims to optimize, rather than the intermediate predictions used to generate them.

p. 2
Where the paper says it

Right: compared with DMD2 and TDM, eDMD trains the fake model on final outputs only (top), draws the queries of intermediate steps from a calibrated density (middle; ticks: the level each step denoises to), and answers low-noise queries with a multi-step teacher instead of a single prediction (bottom; percentages: answers on the tree). model should represent the distribution of the samples that distillation ultimately aims to optimize, rather than the intermediate predictions used to generate them.

The resulting expected parameter update follows a negative gradient pushing queries toward higher log(pτ /qτ).

The authors demonstrate that eDMD improves 4-step generation quality over existing distribution matching methods, achieving the highest peak HPSv3++ scores across all four backbones. The ablation studies confirm that removing the endpoint reference causes the largest overall degradation among component ablations on both SD3-Medium and FLUX. p. 8

Where the paper says it

Endpoint reference. Removing E causes the largest overall degradation among the component ablations on both SD3-Medium and FLUX.

2-klein-4B. p. 1
Where the paper says it

Experiments on SD3-Medium, FLUX. 2-klein (4B and 9B), and Qwen-Image-2.

The precise teacher (PT) and calibrated query density are shown to be crucial for improving the quality of teacher supervision. p. 8
Where the paper says it

We further improve teacher supervision in two ways: a precise teacher (PT) refines low-noise targets through multi-step denoising, and a calibrated query density favors noise levels where teacher predictions are more reliable. Experiments on four diffusion and flow matching backbones show that eDMD improves 4-step generation quality over existing distribution matching methods.

The final results show that eDMD consistently outperforms existing distribution matching baselines on all seven metrics across the tested backbones. p. 2
Where the paper says it

1 [38]), distilling each into a 4-step generator without using real images. Extensive experiments demonstrate that eDMD consistently outperforms existing distribution matching baselines, achieving the highest peak HPSv3++ [26] scores across all four backbones.

The authors show that training the fake model on intermediate predictions or states can distort distribution matching in multi-step DMD. The resulting reference residual can persist even after the student’s final output distribution matches the teacher’s. p. 8

Where the paper says it

Conclusion We showed that training the fake model on intermediate predictions or states can distort distribution matching in multi-step DMD. The resulting reference residual can persist even after the student’s final output distribution matches the teacher’s.

This mismatch is resolved by aligning the fake model’s reference with the endpoint distribution, eliminating the reference residual and ensuring that distribution matching is focused on samples delivered at inference. p. 8
Where the paper says it

eDMD resolves this mismatch with an endpoint reference, training a shared fake model exclusively on final outputs and using it to supervise every denoising step. This keeps distribution matching focused on the samples delivered at inference and gives intermediate predictions the freedom to adapt to few-step generation.

The authors propose eDMD, which trains a single shared fake model exclusively on the student’s final generated outputs and uses it to provide correction signals for every denoising step. The loss function used for training the fake model is defined as LF = E h λ(τ) F((1 − τ)x + τϵ, τ − x / 2i, (7). The resulting expected parameter update follows a negative gradient pushing queries toward higher log(pτ /qτ).

The authors identify and theoretically characterize an objective mismatch in multi-step DMD where training the fake model on intermediate predictions introduces a reference residual that distorts distribution matching and vanishes if and only if the fake model learns the endpoint distribution. This mismatch is characterized by a reference residual R which remains nonzero even when the student’s final output distribution matches the teacher’s, and this residual vanishes if and only if the fake model represents the endpoint distribution.

Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.

Improvements for AI systems

  1. Endpoint Distribution Matching Distillation (eDMD) eliminates reference residual by training a single shared fake model exclusively on student final outputs and using it to correct every denoising step, ensuring the distribution matching signal is determined by the discrepancy between the teacher and the student’s final outputs, independently of the intermediate sampling trajectory.

  2. Teacher supervision is strengthened via a precise teacher (PT) that replaces singlestep guided predictions with short multi-step trajectories on the final noise interval and a calibrated query density to concentrate intermediate-step queries on noise levels providing more reliable supervision.

  3. The system can preserve fine details in complex structures like fur, hair, and grass better than existing methods, achieving higher human-preference scores and fewer structural artifacts across various backbones.

Abstract

Distribution matching distillation (DMD) has emerged as a leading paradigm for distilling diffusion models into few-step students, where an auxiliary fake score model tracks the student's distribution and the discrepancy between the teacher and fake predictions drives the student toward the teacher. In multi-step distillation, although only the student's final output is delivered, existing methods train the fake model on intermediate states or across all denoising steps. Consequently, the student's update signal becomes entangled with intermediate distributions, which can distort the matching direction and remain nonzero even after the student's final outputs already match the teacher distribution. To resolve this mismatch, we propose Endpoint Distribution Matching Distillation (eDMD), which trains a single shared fake model exclusively on the student's final outputs and uses it to correct every denoising step. Our theoretical analysis proves that this endpoint reference guarantees that the distribution matching term vanishes if and only if the student's final output distribution matches the teacher's, regardless of intermediate distributions. Beyond the fake reference, we further strengthen teacher supervision through a precise teacher (PT) that replaces single-step guided predictions with short multi-step trajectories on the final noise interval, paired with a calibrated query density for intermediate steps. Experiments on SD3-Medium, FLUX.2-klein (4B and 9B), and Qwen-Image-2.1 demonstrate that eDMD consistently outperforms existing distribution matching methods for 4-step generation.

Sources

Related papers