DUEL: Exact Likelihood Reshapes Masked Diffusion Scaling Laws and Beyond
cs.LG
Submitted: 2026-03-02
Updated: 2026-11-04
Comments: 36 pages, 4 figures, 21 tables. Code: https://github.com/kuleshov-group/duel. Project page: https://giladturok.github.io/duel/
Project page: https://giladturok.github.io/duel
License: http://creativecommons.org/licenses/by/4.0/
The gist: Masked diffusion models (MDMs) are trained to predict tokens in every order but generate in one.
Terminology
Abstract
Masked diffusion models (MDMs) are trained to predict tokens in every order but generate in one. As such, MDM likelihood is an intractable sum over all orders. The evidence lower bound (ELBO) is used as a proxy for likelihood, but it is a loose bound and assumes positions are unmasked uniformly at random. Frontier MDMs, however, select positions with deterministic rules such as confidence. To resolve this mismatch we introduce DUEL, a framework that unifies common deterministic MDM samplers. DUEL computes the likelihood exactly, under the sampler that is actually used during generation. DUEL is a simple plug-and-play algorithm applicable to any pretrained MDM with no retraining. To compute the likelihood of a text sequence, simply replay generation, reveal the true tokens instead of sampled ones, and sum their log-probabilities. Evaluating with DUEL instead of the ELBO changes what we conclude about MDMs. The perplexity gap between MDMs and autoregressive (AR) models shrinks by up to 32% in-domain and 82% zero-shot. In scaling laws, the reported 16x MDM-AR compute gap falls to 9x under DUEL, showing over 1/3 of it is an evaluation artifact. Scoring multiple-choice answers with likelihood via DUEL instead of the ELBO improves LLaDA-8B accuracy on all six tasks, by up to 9.7 points. DUEL also selects samplers correctly: it matches a gold-standard LLM judge on every decided comparison, which generative perplexity and MAUVE fail to do. DUEL reveals that MDMs are more competitive with AR models than the ELBO suggests.
Sources
- Block Diffusion: Interpolating Between Autoregressive and Diffusion Language Models
- Encoder-Decoder Diffusion Language Models for Efficient Training and Inference
- Leak, Cheat, Repeat: Data Contamination and Evaluation Malpractices in Closed-Source LLMs
- Accelerated Sampling from Masked Diffusion Models via Entropy Bounded Unmasking
- LLaDA2.0: Scaling Up Diffusion Language Models to 100B
- One Billion Word Benchmark for Measuring Progress in Statistical Language Modeling
- dUltra: Ultra-Fast Diffusion Language Models via Reinforcement Learning
- dParallel: Learnable Parallel Decoding for dLLMs
- A Discourse-Aware Attention Model for Abstractive Summarization of Long Documents
- Beyond Autoregression: Fast LLMs via Self-Distillation Through Time
- BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
- Reviving Any-Subset Autoregressive Models with Principled Parallel Sampling and Speculative Decoding
- Distillation of Discrete Diffusion through Dimensional Correlations
- Demystifying MaskGIT Sampler and Beyond: Adaptive Order Selection in Masked Diffusion
- Autoregressive Diffusion Models
- Don't Settle Too Early: Self-Reflective Remasking for Diffusion Language Models
- Learning Unmasking Policies for Diffusion Language Models
- Information-Theoretic Discrete Diffusion
- Train for the Worst, Plan for the Best: Understanding Token Ordering in Masked Diffusions
- KLASS: KL-Guided Fast Inference in Masked Diffusion Models
Related papers
- FlexMoE: Learning One-for-All Multi-Budget Expert Pruning for Flexible MoE Deployment
- OptiMind: Improving Small Language Models for Optimization Modeling through Formulation Guidance and Multi-Turn Learning
- Staying in the Sweet Spot: Boost Reasoning Evolution via Responsive Rollout Difficulty Manipulation
- Examining the Impact of Data and Training Methods on Emergent and Subliminal Misalignment
- Rethinking Black-Box On-Policy Distillation: A Rubric-based Analysis
- (Mis)generalization of Helpful-Only Fine-tuning