What Does a Discrete Diffusion Model Learn? One Reverse Process in Three Coordinates
cs.LG, cs.AI, cs.CL, cs.IT, math.IT, stat.ML
Submitted: 2026-07-06
Updated: 2026-10-14
Comments: 66 pages, 6 figures
License: http://creativecommons.org/licenses/by/4.0/
The gist: What does a discrete diffusion model learn: a denoiser, a score ratio, or a bridge plug-in predictor? At the level of jump rates, these are one object in different coordinates, and reading a neural
Terminology
Abstract
What does a discrete diffusion model learn: a denoiser, a score ratio, or a bridge plug-in predictor? At the level of jump rates, these are one object in different coordinates, and reading a neural network in the wrong coordinate changes the process being trained and sampled. Starting with a rigorous derivation of the continuous-time Markov chain (CTMC) ELBO for any noising process, boundary terms included, we prove the Oracle Distance theorem: the negative ELBO is exactly equal to the data entropy plus the path KL from the oracle reverse process to the learned one, not merely a bound. Its unique optimizer is therefore the conditional expectation of the true reverse jump rate given the current noisy state, and its irreducible cost is the rate at which the forward process Z t destroys information about the clean data Z 0, - dt I(Z 0; Z t), so every noising process shares the same best achievable negative ELBO: the data entropy. For sequences with token-factorizing noise, the oracle projection yields three exact coordinates for the optimizer: denoiser, cavity (bridge plug-in), and score, with closed-form conversions among them. This framework identifies which law each loss in the literature actually optimizes, recovering MDM, UDM, SEDD, and GIDD as special cases; explains why denoiser and cavity coincide for masked diffusion but not for uniform diffusion; proves that a denoiser parameterization makes the uniform ELBO diverge at initialization while the bridge plug-in stays finite; and calibrates ELBO implementations exactly at initialization. Every identity is verified numerically, without approximation, on an exactly solvable model.
Sources
- Uniform Diffusion Models Revisited: Leave-One-Out Denoiser and Absorbing State Reformulation
- Information-Theoretic Discrete Diffusion
- Auto-Encoding Variational Bayes
- Large Language Diffusion Models
- Noise Scheduling as Information-Guided Allocation in Diffusion Training
- Scaling Beyond Masked Diffusion Language Models
- Demystifying Diffusion Objectives: Reweighted Losses are Better Variational Bounds
- A Reparameterized Discrete Diffusion Model for Text Generation
Related papers
- Polynomial-Augmented Neural Networks (PANNs) with Weak Orthogonality Constraints for Enhanced Function and PDE Approximation
- AIRL-S: Unifying Reinforcement Learning and Search-Based Test-Time Scaling via Adversarial Inverse Reinforcement Learning
- Transformers as Bayesian In-Context Experimenters: Smoothness-Adaptive Efficient ATE Estimation
- Convergence issues in Relational Concept Analysis based on AOC-posets
- Beliefs Beyond Posteriors: The Predictive Cost of ELBO Binding
- Understanding Diffusion Models via Ratio-Based Function Approximation with SignReLU Networks