LEAP: Unlocking dLLM Parallelism via Lookahead Early-Convergence Token Detection
cs.LG, cs.AI
Submitted: 2026-05-09
Updated: 2026-10-14
License: http://creativecommons.org/licenses/by/4.0/
The gist: Diffusion Language Models (dLLMs) have garnered significant attention for their potential in highly parallel processing.
Terminology
Abstract
Diffusion Language Models (dLLMs) have garnered significant attention for their potential in highly parallel processing. The parallel capabilities of existing dLLMs stem from the assumption of conditional independence at high confidence levels, which ensures negligible discrepancy between the marginal and joint distributions. However, the stringent confidence thresholds required to preserve accuracy severely constrain the scalability of parallelism. Through systematic token-level statistical analysis, we reveal that a substantial proportion of tokens converge to their correct predictions early in the denoising process yet fail to reach standard confidence thresholds, confirming that current confidence-based criteria are overly conservative. In response, we introduce LEAP (Lookahead Early-Convergence Token Detection for Accelerated Parallel Decoding). LEAP is a training-free, plug-and-play method that leverages future context filtering and multi-sequence superposition to detect early-converging tokens. By validating the alignment between early convergence and correctness, we enable reliable early decoding of these tokens. Benchmarking across diverse domains demonstrates that LEAP significantly lowers inference latency and decoding steps. Compared to confidence-based decoding, the average number of denoising steps is reduced by about 30%. On the GSM8K dataset, combining LEAP with dParallel accelerates decoding to 7.2 tokens per step while preserving model precision. LEAP effectively breaks the reliance on high-confidence priors, offering a novel paradigm for parallel decoding.
Sources
- GPT-4 Technical Report
- Program Synthesis with Large Language Models
- Accelerated Sampling from Masked Diffusion Models via Entropy Bounded Unmasking
- LLaDA2.0: Scaling Up Diffusion Language Models to 100B
- Evaluating Large Language Models Trained on Code
- dParallel: Learnable Parallel Decoding for dLLMs
- Training Verifiers to Solve Math Word Problems
- From Bits to Rounds: Parallel Decoding with Exploration for Diffusion Language Models
- KLASS: KL-Guided Fast Inference in Masked Diffusion Models
- ReFusion: A Diffusion Large Language Model with Parallel Autoregressive Decoding
- Diffusion Language Models Know the Answer Before Decoding
- WeDLM: Reconciling Diffusion Language Models with Standard Causal Attention for Fast Inference
- DeepSeek-V3 Technical Report
- TiDAR: Think in Diffusion, Talk in Autoregression
- dKV-Cache: The Cache for Diffusion Language Models
- Large Language Diffusion Models
- Seed Diffusion: A Large-Scale Diffusion Language Model with High-Speed Inference
- Diffusion LLMs Can Do Faster-Than-AR Inference via Discrete Diffusion Forcing
- Fast-dLLM: Training-free Acceleration of Diffusion LLM by Enabling KV Cache and Parallel Decoding
- LoPA: Scaling dLLM Inference via Lookahead Parallel Decoding
Related papers
- Polynomial-Augmented Neural Networks (PANNs) with Weak Orthogonality Constraints for Enhanced Function and PDE Approximation
- AIRL-S: Unifying Reinforcement Learning and Search-Based Test-Time Scaling via Adversarial Inverse Reinforcement Learning
- Transformers as Bayesian In-Context Experimenters: Smoothness-Adaptive Efficient ATE Estimation
- Convergence issues in Relational Concept Analysis based on AOC-posets
- Beliefs Beyond Posteriors: The Predictive Cost of ELBO Binding
- Understanding Diffusion Models via Ratio-Based Function Approximation with SignReLU Networks