Information-Guided Frontier Decoding: Contextual Utility-Driven Commitment in dMLLMs
cs.CL
Submitted: 2026-08-27
Updated: 2026-10-11
Comments: Accepted to Findings of EMNLP 2026
License: http://creativecommons.org/licenses/by/4.0/
The gist: Decoding quality in diffusion multimodal language models (dMLLMs) depends heavily on the order in which masked tokens are committed.
Terminology
Abstract
Decoding quality in diffusion multimodal language models (dMLLMs) depends heavily on the order in which masked tokens are committed. Existing confidence-based strategies prioritize locally easy tokens, but confidence does not necessarily reflect contextual usefulness. As a result, structurally easy tokens such as punctuation may be committed before informative semantic anchors, weakening context propagation and increasing error accumulation. We propose Information-Guided Frontier Decoding (IGFD), a training-free decoding strategy that ranks candidates using token confidence, neighborhood uncertainty, and structural commitment risk. IGFD encourages early commitment of reliable semantic anchors while delaying fragile structural tokens, improving contextual support during decoding. A dynamic candidate frontier further constrains token selection to locally expandable regions under the same decoding budget. The method requires no additional training, auxiliary models, or extra forward passes. Experiments across multimodal understanding, reasoning, grounding, and hallucination benchmarks show that IGFD consistently outperforms existing decoding strategies across the majority of benchmarks and diffusion MLLM backbones under identical decoding budgets.
Sources
- $R^2$-dLLM: Accelerating Diffusion Large Language Models via Spatio-Temporal Redundancy Reduction
- Locally Confident, Globally Stuck: The Quality-Exploration Dilemma in Diffusion Language Models
- DeCoRe: Decoding by Contrasting Retrieval Heads to Mitigate Hallucinations
- LLaDA2.0: Scaling Up Diffusion Language Models to 100B
- Confidence-Based Decoding is Provably Efficient for Diffusion Language Models
- D$^{3}$ToM: Decider-Guided Dynamic Token Merging for Accelerating Diffusion MLLMs
- Mitigating Mask Prior Drift and Positional Attention Collapse in Large Diffusion Vision-Language Models
- Beyond Confidence: Adaptive and Coherent Decoding for Diffusion Language Models
- Empirical Analysis of Decoding Biases in Masked Diffusion Models
- Train for the Worst, Plan for the Best: Understanding Token Ordering in Masked Diffusions
- Semantic Uncertainty: Linguistic Invariances for Uncertainty Estimation in Natural Language Generation
- Lookahead Unmasking Elicits Accurate Decoding in Diffusion Language Models
- Diffusion Language Models Know the Answer Before Decoding
- AdaBlock-dLLM: Semantic-Aware Diffusion LLM Inference via Adaptive Block Size
- Pointer Sentinel Mixture Models
- Flexible and Efficient Grammar-Constrained Decoding
- Deferred Commitment Decoding for Diffusion Language Models
- Accelerating Diffusion Large Language Models with SlowFast Sampling: The Three Golden Principles
- Fast-dLLM: Training-free Acceleration of Diffusion LLM by Enabling KV Cache and Parallel Decoding
- WavefrontDiffusion: Dynamic Decoding Schedule for Improved Reasoning
Related papers
- Exploring Solution Divergence and Its Effect on Large Language Model Problem Solving
- Ishigaki-IDS-Bench: A Benchmark for Generating Information Delivery Specification from BIM Information Requirements
- Subliminal Steering: Stronger Encoding of Hidden Signals
- MedStruct-S: A Benchmark for Key Discovery, Key-Conditioned QA and Semi-Structured Extraction from OCR Clinical Reports
- The End of Transformers? On Challenging Attention and the Rise of Sub-Quadratic Architectures
- Untangling the Mechanisms of Misleading Context in Medical Question Answering