Improved Sampling Schedules for Discrete Diffusion Models
Listen
Radio episode about this paper
Transcript
Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.
Tom: We're looking at the paper: "Improved Sampling Schedules for Discrete Diffusion Models".
Jane: Here's the gist of the paper. Discrete diffusion models have emerged as a powerful paradigm for generative modeling on sequence data; however, the information-theoretic principles governing their reverse processes remain significantly less understood than those of their continuous counterparts.
Tom: In this work, we bridge this gap by analyzing the reverse process dynamics through the lens of thermodynamic entropy production.
Jane: We propose the entropy production rate as a rigorous proxy for quantifying information generation, deriving as a byproduct a bound on the Wasserstein distance between intermediate states and the data distribution.
Tom: Leveraging these insights, we introduce two novel sampling schedules that are uniformly spaced with respect to their corresponding physics-inspired metrics: the Entropic Discrete Schedule (EDS), which is defined by maintaining a constant rate of information gain, and the Wasserstein Discrete Schedule (WDS), which is defined by taking equal steps in terms of the Wasserstein distance.
Title and authors: Tom: So we're looking at this paper today, "Improved Sampling Schedules for Discrete Diffusion Models." It sounds like they're tackling a big problem with how we generate data from these discrete diffusion models.
Jane: Exactly. They’ve been saying that the information-theoretic principles behind how these models reverse their process are still really unclear compared to continuous ones, and this paper is trying to bridge that gap by using thermodynamic entropy production to look at the dynamics.
Lu: From my side, I'm interested in how they use entropy production as a way to quantify information generation, because that's the missing piece of understanding for these reverse processes.
Meng: Okay, so they propose two new sampling schedules called the Entropic Discrete Schedule or EDS and the Wasserstein Discrete Schedule or WDS. What do those actually mean in practice for someone trying to run a model?
Tom: Well, what they're saying is that these schedules are designed to be uniform based on physics-inspired metrics, which means they aren't just picking random steps. The Entropic Discrete Schedule aims to keep the rate of information gain constant throughout the sampling process.
Jane: And the Wasserstein Discrete Schedule is defined by taking equal steps in terms of how far apart those intermediate states are when measured by a specific metric called the Wasserstein distance. That sounds like they're making the sampling trajectory follow some kind of physical path through the data space.
Lu: The paper argues that this thermodynamic approach gives them a rigorous proxy for quantifying information generation, which they then use to derive a bound on the Wasserstein distance between intermediate states and the actual data distribution > <ref:2602.06849#pg2>.
Meng: That sounds promising for stability, but what about when things get tricky? They mention that for non-reversible processes, like absorbing diffusion where the model gets stuck in a mask state, the standard definition of total entropy production can diverge to infinity > <ref:2602.06849#pg3>.
Jane: Right, that's a huge caveat. The authors address this by using an adiabatic–non-adiabatic decomposition of entropy production to get a finite quantity that works across different discrete processes > <ref:2602.06849#pg3>.
Title and authors: Tom: So they derive the non-adiabatic entropy production rate, H na(theta)(t), which is then integrated over the generation interval from zero to T to get the total non-adiabatic entropy production, H na(theta) = - integral zero T E x about p t(x) t"X y from p t(x) Q t(x, y) s p theta(x, y) dt > <ref:2602.06849#pg1> <ref:2602.06849#pg3>.
Lu: That integral expression is the specific way they estimate it using a neural network based on the learned score function, p theta, and it requires no additional training for that part of the estimation > <ref:2602.06849#pg3>.
Meng: From an engineering standpoint, if you're going to use a neural network to estimate this rate, you need to know how stable that estimator is when deployed at scale. What are the practical implications of using this entropy production measure instead of just standard metrics?
Jane: It provides a way to find a time discretization where the progress between consecutive steps is uniform, by inverting some warping function which represents the normalized cumulative progress > Page five.
Tom: So they introduce two schedules based on that concept: the Entropic Discrete Schedule or EDS, which keeps a constant rate of information gain, and the Wasserstein Discrete Schedule or WDS, which takes equal steps in terms of the Wasserstein distance > <ref:2602.06849#pg1>.
Lu: The main point they make is that these schedules should be derived from the thermodynamic properties of the generative process itself rather than using some heuristic functions to pick a time schedule > Page five.
Meng: So, what's the practical payoff for a production system? Does this mean faster inference or better quality in real-world scenarios?
Jane: The results show that both EDS and WDS lead to significant efficiency gains, achieving high-fidelity generation with much fewer function evaluations than standard uniform baselines > Page sixteen.
Tom: And WDS specifically was particularly robust, consistently achieving the lowest perplexity and surpassing existing error-based schedules in text generation and language modeling > Page fourteen.
Lu: The empirical demonstrations are impressive, especially seeing WDS perform better on text generation across different model scales > Page sixteen.
Title and authors: Meng: I'm looking at the data here, and for count data, WDS achieved a four times inference speedup compared to uniform schedules in the low-NFE regime of two to sixteen stpdf > Table one <ref:2602.06849#pg1>. That kind of speedup is tangible when you're dealing with large models.
Jane: That’s a really concrete number that shows the practical benefit, and it suggests that aligning sampling with these intrinsic dynamics actually translates into real performance improvements across different modalities > Page sixteen.
Tom: It confirms what they were suggesting: the "physical time" of a diffusion process rarely aligns perfectly with its "semantic progress" > Conclusion.
Lu: What this means for the broader field is that we can now use these thermodynamic principles to derive principled improvements to sampling strategies, which was previously only explored in continuous diffusion models > Page two <ref:2602.06849#pg1>.
Meng: For deployment, it means we can find a plug-and-play solution that significantly enhances inference efficiency without needing a completely new training pipeline for the model itself > Conclusion.
Jane: So, we've seen how deriving schedules from entropy production leads to tangible gains in speed and quality across music notation, vision, and language modeling > Page fourteen.
Tom: It really comes down to moving away from guessing the schedule and instead basing it on the model's own internal information flow dynamics > Conclusion.
Lu: And for future work, I think exploring how these concepts apply to more complex sequence tasks beyond what they tested could be a good direction > Conclusion.
Meng: If we can use this thermodynamic framework to find a stable proxy for the Wasserstein distance bound using only non-adiabatic entropy production, that would make measuring geometric evolution in probability space numerically more stable > Conclusion.
Jane: So, to wrap up on "Improved Sampling Schedules for Discrete Diffusion Models," it seems like aligning the sampling trajectory with these intrinsic dynamics through entropy production is a powerful way to improve inference efficiency and quality across discrete diffusion models > Conclusion.
The paper's summary: Tom: So, to recap, this paper is about taking discrete diffusion models—the ones that work on things like text or music—and fixing how they sample them during generation by using ideas from thermodynamics, specifically entropy production.
Jane: Right. Instead of just picking random steps for the reverse process, the authors are proposing two specific schedules: one based on keeping the information gain constant, which they call EDS, and another based on moving through space in terms of Wasserstein distance > <ref:2602.06849#pg1>.
Lu: What's interesting here is that they argue you shouldn't just use some random heuristic for how fast you sample; the schedule should come from the actual physics of the generative process itself > Page five.
Meng: So, what’s the big deal for me? If I’m trying to deploy this, what does "information gain" or "Wasserstein distance" actually mean in my deployment pipeline?
Tom: Well, they've built a way to measure the non-adiabatic entropy production using a neural network that doesn't require extra training, and they use that measurement to define the schedule > Page thirteen.
Jane: They claim this thermodynamic measurement gives them a stable proxy for how far apart intermediate states are in probability space, which is more reliable than just looking at the total entropy production when things get tricky with absorbing states > <ref:2602.06849#pg4>.
Lu: That’s the technical angle, but look at the results they show. They test these schedules on different things like synthetic data, music notation, vision and language modeling > Page fourteen.
Meng: And what did they actually find in those tests? Did it just make things slightly better, or is it a real efficiency gain?
Tom: It’s a real efficiency gain. They show that both EDS and WDS beat the standard uniform sampling schedules across the board, achieving high quality generation with much fewer function evaluations > Page sixteen.
Jane: And for text generation specifically, the Wasserstein Discrete Schedule, or WDS, was particularly robust; it consistently got lower perplexity than other error-based schedules > Page fourteen.
Meng: I saw a number in Table one that really stuck with me. For count data, the WDS schedule gave them a four times inference speedup compared to uniform schedules when the noise level, or NFE, was low—between two and sixteen stpdf > Table one. That’s tangible performance for production systems.
Lu: So the main idea is that by aligning your sampling trajectory with these intrinsic dynamics of the model, you get a plug-and-play solution that improves inference efficiency without needing to retrain the model itself > Conclusion.
Jane: It really moves us away from guessing what schedule to use and toward basing it on how the model naturally flows through its data space > Conclusion.
Tom: The paper shows that the "physical time" of a diffusion process doesn't always match its "semantic progress," and this scheduling method realigns them > Conclusion.
Lu: And they suggest using this framework to get a stable metric for geometric evolution in probability space, which is useful for understanding these processes deeper > Conclusion.
Meng: So it sounds like if you’re running these models, you could switch to WDS or EDS and see a real speedup and better quality, provided you can handle the setup of that entropy estimation network > Conclusion.
The paper's improvements: Tom: So, moving on from the setup, they’re proposing these new schedules—EDS and WDS—which are meant to make sure the time steps in the reverse process are uniform based on some physical property rather than just a guess > Page five.
Jane: Right. The main improvement is that instead of picking steps randomly, you tailor them so that the rate of information gain stays constant for EDS, and you take equal steps in terms of Wasserstein distance for WDS > <ref:2602.06849#pg1>.
Lu: What they’re really pushing is that this schedule should come from the actual dynamics of the generative process itself, not just some heuristic function we throw in > Page five.
Meng: So if I’m running these models at scale, what does this mean for my deployment pipeline? Does it just mean less training time or something more practical about how fast it runs?
Tom: It means they show that both schedules lead to high-fidelity generation with a lower computational budget than the old uniform methods > Page sixteen.
Jane: And WDS specifically showed this robustness, consistently getting the lowest perplexity in text generation across different model sizes > Page fourteen.
Meng: I saw something in Table one about count data where WDS gave them a four times inference speedup compared to uniform schedules when the noise level was low—between two and sixteen stpdf > Table one. That’s a concrete number for deployment considerations.
Lu: The core idea is that by using thermodynamic principles, you get a better way to measure geometric evolution in probability space using non-adiabatic entropy production, which makes the measurement more stable > Page thirteen.
Jane: It seems like they’re giving us a way to find where the progress is happening naturally within the model's structure > Conclusion.
Tom: They're essentially saying that the "physical time" of a diffusion process doesn't always match its "semantic progress," and by realigning the sampling trajectory, you get better results > Conclusion.
Lu: The implication is we can use this framework to derive principled improvements to sampling strategies, which was usually only explored in continuous diffusion models > Page two.
Jane: So, for someone just listening to the show, it means these schedules are a way to make the AI run faster and produce higher quality outputs without needing a totally new training pipeline > Conclusion.
Meng: The limitation they mention is that they rely on estimating this non-adiabatic entropy production with a neural network, which adds complexity to the initial setup of the schedule > Page thirteen.
Tom: That’s fair. So it’s not just plug-and-play for everything, but it's a way to get significantly better results when you can afford that little extra step in the initial setup > Conclusion.
Conclusion: Tom: So, to wrap things up on "Improved Sampling Schedules for Discrete Diffusion Models," we've seen how using thermodynamic entropy production—the non-adiabatic rate—can give us a rigorous way to define sampling schedules like EDS and WDS > Page thirteen.
Jane: Basically, the authors show that you can derive these schedules from the process's own physics instead of just guessing what steps to take > Conclusion.
Lu: The big implication is that this gives us a plug-and-play solution to significantly enhance inference efficiency across discrete diffusion models without having to retrain the model itself > Conclusion.
Meng: From my side, it means if I’m running these models in production, I can find a way to make them run faster and better quality than the standard uniform schedules we use now > Conclusion.
Tom: Exactly. They prove that aligning sampling with the intrinsic dynamics of the model leads to much better performance at a lower computational budget > Conclusion.
Jane: It really comes down to moving away from guessing how long to sample and basing it on what's happening inside the model's information flow > Conclusion.
Lu: And they flag that while this is a great method, it does require estimating that entropy production rate with a neural network, which adds an initial layer of complexity > Page thirteen.
Meng: Yeah, so it’s not totally zero effort to implement; you still have to set up the estimator properly before you can get those speedups on your traffic > Conclusion.
Lalam: I'm confident in this direction; the stability provided by using non-adiabatic entropy production as a proxy for Wasserstein distance is solid and reliable > Conclusion.
Tom: Alright, so we’ve covered how they use that rate to define EDS and WDS, and the real numbers show they beat baselines across music notation, vision, and language modeling > Page fourteen.
Jane: It’s clear that aligning the sampling trajectory with these intrinsic dynamics is a powerful way to get tangible gains in speed and quality > Conclusion.
Lu: Moving forward, we should look at how these thermodynamic concepts apply to even more complex sequence tasks beyond what they tested here > Conclusion.
Department of Data Science, EURECOM · Donders Institute for Brain, Cognition and Behaviour
cs.LG
Submitted: 2026-02-06
Updated: 2026-10-29
License: http://creativecommons.org/licenses/by/4.0/
Importance score: 80/100
The gist: Discrete diffusion models have emerged as a powerful paradigm for generative modeling on sequence data; however…
Key concepts
- Entropy Production Rate (Htot)
- This rate quantifies the irreversibility of the diffusion process, measuring how much disorder is generated during the reverse path. It is used as a rigorous proxy for quantifying information generation in discrete models.
- Entropic Discrete Schedule (EDS)
- This sampling strategy adjusts the time steps based on information load. It ensures a constant rate of information gain throughout the generation process, effectively slowing down during high-information phases and speeding up during low-information ones.
- Wasserstein Discrete Schedule (WDS)
- This schedule bases time steps on the transport velocity within the Wasserstein metric, which measures geometric distance between probability distributions. It adapts sampling to the speed of movement in this metric, proving particularly effective for count data.
Terminology
Summary
Discrete diffusion models have emerged as a powerful paradigm for generative modeling on sequence data; however, the information-theoretic principles governing their reverse processes remain significantly less understood than those of their continuous counterparts. p. 0 Improved Sampling Schedules for Discrete Diffusion Models
Alberto Foresti * 1 Mustapha Bounoua * 1 Giulio Franzese 1 Luca Ambrogioni 2 Pietro Michiardi 1
Abstract
Discrete diffusion models have emerged as a powerful paradigm for generative modeling on sequence data; however, the information-theoretic
principles governing their reverse processes remain significantly less understood than those of
their continuous counterparts. In this work, we
bridge this gap by analyzing the reverse process
dynamics through the lens of thermodynamic entropy production. Improved Sampling Schedules for Discrete Diffusion Models
Alberto Foresti * 1 Mustapha Bounoua * 1 Giulio Franzese 1 Luca Ambrogioni 2 Pietro Michiardi 1
Abstract
Discrete diffusion models have emerged as a powerful paradigm for generative modeling on sequence data; however, the information-theoretic
principles governing their reverse processes remain significantly less understood than those of
their continuous counterparts. In this work, we
bridge this gap by analyzing the reverse process
dynamics through the lens of thermodynamic entropy production. In this work, we
bridge this gap by analyzing the reverse process
dynamics through the lens of thermodynamic entropy production. We propose the entropy production rate as a rigorous proxy for quantifying
information generation, deriving as a byproduct
a bound on the Wasserstein distance between intermediate states and the data distribution. We propose the entropy production rate as a rigorous proxy for quantifying
information generation, deriving as a byproduct
a bound on the Wasserstein distance between intermediate states and the data distribution. Leveraging these insights, we introduce two novel sampling schedules that are uniformly spaced with
respect to their corresponding physics-inspired
metrics: the Entropic Discrete Schedule (EDS),
which is defined by maintaining a constant rate
of information gain, and the Wasserstein Discrete
Schedule (WDS), which is defined by taking equal
steps in terms of the Wasserstein distance. Leveraging these insights, we introduce two novel sampling schedules that are uniformly spaced with
respect to their corresponding physics-inspired
metrics: the Entropic Discrete Schedule (EDS),
which is defined by maintaining a constant rate
of information gain, and the Wasserstein Discrete
Schedule (WDS), which is defined by taking equal
steps in terms of the Wasserstein distance. We empirically demonstrate that our proposed schedules
significantly outperform state-of-the-art strategies
across diverse application domains, including synthetic data, music notation, vision and language
modeling, consistently achieving superior performance at a lower computational budget.Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Thermodynamic Foundation of Information Flow
The reverse generative trajectory can be analyzed through the lens of stochastic thermodynamics. The irreversibility of the diffusion process is quantified by the Entropy Production rate Htot. p. 1 Entropy Production
The reverse generative trajectory can be analyzed through
the lens of stochastic thermodynamics. In this framework,
the irreversibility of the diffusion process is quantified by
the Entropy Production rate Htot. In this framework,
the irreversibility of the diffusion process is quantified by
the Entropy Production rate Htot. Following the standard
formulation (Seifert, 2005; Esposito & Van den Broeck,
2010; Premkumar, 2024), this is defined as the integral of
the log ratio divergence between the backward path and the
time-reversal of the forward path (See Section A. In uniform diffusion, the stationary distribution πt ≡ π is
uniform, and detailed balance holds. Then, the adiabatic
contribution vanishes (Had = 0), and the total entropy production is entirely captured by the non-adiabatic term. In contrast, for masked (absorbing) diffusion, the stationary distribution concentrates on the absorbing state, and
detailed balance is violated. In this setting, the adiabatic
entropy production diverges due to irreversible probability
currents into the absorbing state, reflecting the infinite housekeeping cost of maintaining the absorbing prior. In this setting, the adiabatic
entropy production diverges due to irreversible probability
currents into the absorbing state, reflecting the infinite housekeeping cost of maintaining the absorbing prior. Crucially,
the non-adiabatic entropy production Hna remains finite,
provided the marginal distributions ←
p t
retain full support
over non-absorbing states.Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Derivation of Sampling Schedules
The efficacy of discrete diffusion sampling depends critically on the time schedule, which dictates the discretization grid for the reverse process. We propose that the improved time schedule should be derived not from heuristic functions, but from the thermodynamic properties of the generative process itself. p. 3 Such aggressive discretization in high-entropy regions may
yield approximation errors, leading to mode collapse (Park
et al. We propose that the improved time schedule
should be derived not from heuristic functions, but from the
thermodynamic properties of the generative process itself.Where the paper says it
-
Entropic Discrete Schedule (EDS), which adapts to the information load, and
-
Wasserstein Discrete Schedule (WDS), which adapts to the transport velocity in the Wasserstein metric. p. 3
Where the paper says it
Consequently, we propose a physically motivated scheduling approach, where the time discretization is governed by the intrinsic progress of the generative process. We introduce two sampling strategies: the Entropic Discrete Schedule (EDS), which adapts to the information load, and the Wasserstein Discrete Schedule (WDS), which adapts to the transport velocity in the Wasserstein metric.
Quantifying Progress via Intrinsic Metrics
To quantify the progression of the diffusion process, we define a Cumulative Progress function C(t). The objective of our improved scheduling is to find a time discretization where the progress between consecutive steps is uniform. p. 4 Improved Sampling Schedules for Discrete Diffusion Models
The objective of our improved scheduling is to find a time
discretization {tk}
K
k=0 such that the progress between consecutive steps is uniform. We define the Warping Function
Φ : [0, T] → [0, 1] as the normalized cumulative progress:
Φ(t) = C(t)/C(T). Improved Sampling Schedules for Discrete Diffusion Models
The objective of our improved scheduling is to find a time
discretization {tk}
K
k=0 such that the progress between consecutive steps is uniform. We define the Warping Function
Φ : [0, T] → [0, 1] as the normalized cumulative progress:
Φ(t) = C(t)/C(T). Improved Sampling Schedules for Discrete Diffusion Models
The objective of our improved scheduling is to find a time
discretization {tk}
K
k=0 such that the progress between consecutive steps is uniform. We define the Warping Function
Φ : [0, T] → [0, 1] as the normalized cumulative progress:
Φ(t) = C(t)/C(T). (10)
Physically, Φ(t) represents the ratio of the total budget (information generated or distance traversed) that has been
consumed by time t. The improved schedule is obtained
by inverting this warping function. (10)
Physically, Φ(t) represents the ratio of the total budget (information generated or distance traversed) that has been
consumed by time t. The improved schedule is obtained
by inverting this warping function.Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Empirical Performance Across Modalities
We evaluate our proposed sampling schedules, EDS and WDS, across various datasets and models. Our results show that both EDS and WDS demonstrate significant efficiency gains, achieving high-fidelity generation at a fraction of the function evaluations required by uniform baselines. p. 7 Empirically, our method requires no additional training and
generalizes across diverse modalities. Both EDS and WDS
demonstrated significant efficiency gains, achieving highfidelity generation at a fraction of the function evaluations
required by uniform baselines. Both EDS and WDS
demonstrated significant efficiency gains, achieving highfidelity generation at a fraction of the function evaluations
required by uniform baselines. Notably, WDS proved particularly robust, consistently achieving the lowest perplexity
and surpassing existing error-based schedules.Where the paper says it
Where the paper says it
Technical Estimation Methodology
To estimate the non-adiabatic entropy production Hnaθ(t) using a neural network, we express the probability ratio in terms of the learned score. The instantaneous estimator is given by Hnaθ(t) = Xx,y←p t(x)←Qt(x, y)− ln s pθ(x, y) + ln πt(y)πt(x). p. 2 For a CTMC with time-dependent generator
→
Qt
, the corresponding entropy production rates read
Had(t) = X
x,y
←
p t
(x)
←
Qt
(x, y) ln
←
Qt
(x, y)πt(x)
←
Qt
(y, x)πt(y)
, (2)
Hna(t) = X
x,y
←
p t
(x)
←
Qt
(x, y) ln
←
p t
(x)πt(y)
←
p t
(y)πt(x)
. (3)
Here, πt denotes the instantaneous stationary distribution
of the generator
→
Qt
, defined by the invariance condition
πt
→
Qt = 0 (Esposito & Van den Broeck, 2010). , 2024) can be computed as:
Hna
θ
(t) = −
X
x,y
←
p t
(x)
←
Qt
(x, y) ln s
p
θ
(x, y). (7)
The total non-adiabatic entropy generated over the full trajectory is obtained by integrating this rate over time. , 2024) can be computed as:
Hna
θ
(t) = −
X
x,y
←
p t
(x)
←
Qt
(x, y) ln s
p
θ
(x, y). (7)
The total non-adiabatic entropy generated over the full trajectory is obtained by integrating this rate over time. To address this, we derived improved schedules by uniformly sampling timesteps within
the intrinsic space of non-adiabatic entropy or Wasserstein
distance, rather than in linear time. Empirically, our method requires no additional training and
generalizes across diverse modalities.Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Conclusion on Efficiency
Ultimately, our findings confirm that the “physical time” of a diffusion process rarely aligns with its “semantic progress”. By realigning the sampling trajectory to the model’s intrinsic dynamics, we offer a plug-and-play solution that significantly enhances the inference efficiency of discrete diffusion models. p. 7 Ultimately, our findings confirm that the “physical time” of
a diffusion process rarely aligns with its “semantic progress. ”
By realigning the sampling trajectory to the model’s intrinsic dynamics, we offer a plug-and-play solution that
significantly enhances the inference efficiency of discrete
diffusion models.Where the paper says it
--- Page 1 ---
Improved Sampling Schedules for Discrete Diffusion Models Alberto Foresti Mustapha Bounoua Giulio Franzese Luca Ambrogioni Pietro Michiardi 1 Department of Data Science, EURECOM, 450 Route des Chappes, 06410 Biot, France 2 Donders Institute for Brain, Cognition and Behaviour, Radboud University Postbus 9102, 6500 HC Nijmegen, Netherlands Correspondence to Alberto Foresti Mustapha Bounoua Giulio Franzese Pietro Michiardi Preprint February 10, 2026 arXiv:2602.06849v2 [cs. p. 0 (2025) introduced
a distillation framework wherein a few-step student sampler
1
arXiv:2602. 06849v2 [cs.Where the paper says it
--- Page 5 ---
Improved Sampling Schedules for Discrete Diffusion Models The objective of our improved scheduling is to find a time discretization where the progress between consecutive steps is uniform. We define the Warping Function Φ: [0, T] → [0, 1] as the normalized cumulative progress. The improved schedule is obtained by inverting this warping function.
--- Page 13 ---
Improved Sampling Schedules for Discrete Diffusion Models A.5 Derivation of the Non adiabatic Entropy Estimator To estimate the non-adiabatic entropy production Hnaθ(t) using a neural network, we express the probability ratio in terms of the learned score. p. 3 Estimation of Neural Entropy
We consider the reverse process to be simulated using a
neural network trained to approximate the probability ratio s
p
θ
(x, y) (see Section 2). We aim to estimate the nonadiabatic neural entropy Hna
θ
as a proxy for the information
produced by the network during the generative process. For a CTMC with time-dependent generator
→
Qt
, the corresponding entropy production rates read
Had(t) = X
x,y
←
p t
(x)
←
Qt
(x, y) ln
←
Qt
(x, y)πt(x)
←
Qt
(y, x)πt(y)
, (2)
Hna(t) = X
x,y
←
p t
(x)
←
Qt
(x, y) ln
←
p t
(x)πt(y)
←
p t
(y)πt(x)
. (3)
Here, πt denotes the instantaneous stationary distribution
of the generator
→
Qt
, defined by the invariance condition
πt
→
Qt = 0 (Esposito & Van den Broeck, 2010).Where the paper says it
Where the paper says it
--- Page 16 ---
Improved Sampling Schedules for Discrete Diffusion Models Figure 9 CIFAR-10 generation results across different sampling schedules Uniform (left), EDS (center), and WDS (right) Each row shows samples generated at varying NFEs, demonstrating the efficiency and quality trade-offs for each scheduling method.
--- Page 14 ---
Improved Sampling Schedules for Discrete Diffusion Models Table 1 Parameters used for estimating the neural non-adiabatic entropy across different datasets Dataset Samples (N) Timestpdf (Nt) Binomial Distribution 1024 1024 Count Dataset 1024 1024 Monophonic Music 64 1024 CIFAR-10 64 1024 Language Modeling
--- Page 8 ---
Improved Sampling Schedules for Discrete Diffusion Models Figure 5 CIFAR-10 generation results across different sampling schedules Uniform (left), EDS (center), and WDS (right) Each row shows samples generated with different NFEs, demonstrating the efficiency and quality trade-offs for each scheduling method.
Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.
Improvements for AI systems
-
Improved inference speed for discrete diffusion models by implementing Entropic Discrete Schedule (EDS) when information gain is high and Wasserstein Discrete Schedule (WDS) when geometric transport velocity is high. EDS
slows down time during phase transitions where information gain is high and speeds up during low-information regimes, ensuring a constant information rate.
-
Enhanced generation quality across diverse modalities by aligning the sampling trajectory with the intrinsic dynamics of the generative process, leading to superior performance at a lower computational budget. For count data, WDS achieved
a 4x inference speedup
compared to Uniform schedules in the low-NFE regime (2–16 stpdf). -
Better handling of complex sequence generation tasks like monophonic music and language modeling by utilizing schedules derived from thermodynamic principles rather than heuristic functions. WDS consistently achieved the
lowest generative perplexity
on text generation across different model scales. -
Derivation of a stable proxy for the Wasserstein distance bound using only non-adiabatic entropy production, which provides a
numerically more stable
metric for measuring geometric evolution in probability space. This allows for practical computation of the speed limit without relying on potentially divergent total entropy production in absorbing processes.
Abstract
Uniform Discrete Diffusion Models have emerged as a powerful paradigm for generative modeling on sequence data; however, the information-theoretic principles governing their reverse processes remain underexplored. In this work, we bridge this gap by analyzing the reverse process dynamics through the lens of thermodynamic entropy production and dynamical activity. We use a variational expression of the discrete Wasserstein distance to characterize the dynamics of the generation process. Leveraging these insights, we first introduce the Entropic Discrete Schedule (EDS), which aligns with entropy production, and later construct the Wasserstein Discrete Schedule (WDS), which combines entropy production and dynamical activity to act as a geometric cost. Our experimental results on language modeling show that WDS provide the same results as uniform while using up to half of the computations on the MBPP benchmark. The code is available at https://github.com/AlbertoForesti/impscheds.
Sources
- Accelerated Sampling from Masked Diffusion Models via Entropy Bounded Unmasking
- Multi-modal Latent Diffusion
- Diffusion Models For Multi-Modal Generative Modeling
- On the Importance of Noise Scheduling for Diffusion Models
- INFO-SEDD: Continuous Time Markov Chains as Scalable Information Metrics Estimators
- MINDE: Mutual Information Neural Diffusion Estimation
- Learnable Sampler Distillation for Discrete Diffusion Models
- Train for the Worst, Plan for the Best: Understanding Token Ordering in Masked Diffusions
- DiffWave: A Versatile Diffusion Model for Audio Synthesis
- Derivative-Free Guidance in Continuous and Discrete Diffusion Models with Soft Value-Based Decoding
- Unlocking Guidance for Discrete State-Space Diffusion and Flow Models
- Neural Entropy
- Fast Solvers for Discrete Diffusion Models: Theory and Applications of High-Order Algorithms
- Caduceus: Bi-Directional Equivariant Long-Range DNA Sequence Modeling
- Simple Guidance Mechanisms for Discrete Diffusion Models
- Entropic Time Schedulers for Generative Diffusion Models
- Score-based Continuous-time Discrete Diffusion Models
- Informed Correctors for Discrete Diffusion Models
Related papers
- A Physics-Regulated Neural Framework for Learning 3D Grain Growth Dynamics
- FlexMoE: Learning One-for-All Multi-Budget Expert Pruning for Flexible MoE Deployment
- OptiMind: Improving Small Language Models for Optimization Modeling through Formulation Guidance and Multi-Turn Learning
- Staying in the Sweet Spot: Boost Reasoning Evolution via Responsive Rollout Difficulty Manipulation
- Examining the Impact of Data and Training Methods on Emergent and Subliminal Misalignment
- Rethinking Black-Box On-Policy Distillation: A Rubric-based Analysis