Lost in Interpolation: Why Predictive Feedback Fails in Diffusion Language Models
Lavanya Nigam, Ishaan Bansal, Aryan Sood, Vidit Aggarwal, Gaurav Kumar Nayak
cs.CL
Submitted: 2026-08-06
Updated: 2026-08-10
Comments: 15 pages
License: http://creativecommons.org/licenses/by/4.0/
The gist: Soft-masking accelerates the convergence of Masked Diffusion Language Models (MDLMs).
Terminology
Abstract
Soft-masking accelerates the convergence of Masked Diffusion Language Models (MDLMs). Existing formulations build this blend with linear interpolation (LERP) in the raw embedding space, which implicitly treats that space as Euclidean. We analyze the embedding space of MDLMs and find that the mask and predicted-token embeddings maintain a near-constant angle of (about 73) throughout training, while embedding norms remain essentially flat across vocabulary-frequency rank. These indicate a hyperspherical geometry, for which LERP is the wrong interpolation primitive. We introduce Spherical Soft-Masking (S-SM), a drop-in replacement that aggregates the top-(k) predictions with a Fr'echet mean on the hypersphere and blends this mean with the mask direction using spherical linear interpolation (SLERP), then restores the native mask norm. We evaluate S-SM on continued pre-training of a released 169M-parameter MDLM checkpoint across a wide range of inference-time step budgets, SLERP feedback avoids the training degradation that LERP feedback induces and delivers MAUVE gains of up to 2x over the vanilla MDLM baseline and 27.5-56.1% over TopK/LERP at various sampling budgets, alongside consistently lower generative perplexity (16.9-19.6% over the baseline), while leaving output entropy and convergence essentially unchanged.
Sources
- DSFT: Inspiring Diffusion Large Language Models to Comprehend Mathematical and Logical Patterns
- DMax: Aggressive Parallel Decoding for dLLMs
- Language Modeling with Hyperspherical Flows
- Continuous diffusion for categorical data
- Harmonizing Geometry and Uncertainty: Diffusion with Hyperspheres
- Discrete Stochastic Localization for Non-autoregressive Generation
- MDPO: Overcoming the Training-Inference Divide of Masked Diffusion Language Models
- On the Role of Discreteness in Diffusion LLMs
- Context-Aware Initialization for Reducing Generative Path Length in Diffusion Language Models
- Generalized Interpolating Discrete Diffusion
Related papers
- Exploring Solution Divergence and Its Effect on Large Language Model Problem Solving
- Ishigaki-IDS-Bench: A Benchmark for Generating Information Delivery Specification from BIM Information Requirements
- Subliminal Steering: Stronger Encoding of Hidden Signals
- MedStruct-S: A Benchmark for Key Discovery, Key-Conditioned QA and Semi-Structured Extraction from OCR Clinical Reports
- The End of Transformers? On Challenging Attention and the Rise of Sub-Quadratic Architectures
- Untangling the Mechanisms of Misleading Context in Medical Question Answering