Visual Information-Guided Parallel Decoding for Diffusion Multimodal Large Language Models
cs.CV, cs.CL
Submitted: 2026-08-27
Updated: 2026-08-27
License: http://creativecommons.org/licenses/by/4.0/
The gist: Diffusion multimodal large language models (dMLLMs) have recently emerged as a new decoding paradigm for multimodal generation.
Terminology
Abstract
Diffusion multimodal large language models (dMLLMs) have recently emerged as a new decoding paradigm for multimodal generation. Starting from a fully masked sequence, dMLLMs progressively decode the sequence by unmasking a subset of the remaining masked positions at each step. Since the selected tokens serve as the prediction context for subsequent steps, deciding which tokens to decode is crucial to the quality of the final output. The most common strategy prioritizes tokens based on a certainty measure that tends to favor tokens frequently observed in the training data. Recent approaches instead order tokens according to their influence on subsequent predictions, but do not explicitly account for the input image. We propose the Visual Information-Guided Sampler (VIG-Sampler), which prioritizes tokens based on their attention to image tokens. We further impose a constraint that penalizes candidate tokens whose image-attention distributions are similar to those of previously selected tokens, thereby increasing the information gain of the decoded subset. Extensive experiments on 7 captioning and VQA benchmarks with 3 open-source dMLLMs demonstrate the effectiveness of VIG-Sampler, which outperforms the Info-Gain Sampler by an average of 19.3 CIDEr points across the captioning benchmarks and surpasses it on COCO Caption while using only half as many decoding steps.
Sources
- Learning to Parallel: Accelerating Diffusion Large Language Models via Learnable Parallel Decoding
- Benchmarking and Improving Detail Image Caption
- From Bits to Rounds: Parallel Decoding with Exploration for Diffusion Language Models
- Mitigating Mask Prior Drift and Positional Attention Collapse in Large Diffusion Vision-Language Models
- Empirical Analysis of Decoding Biases in Masked Diffusion Models
- Learning Unmasking Policies for Diffusion Language Models
- Thinking Diffusion: Penalize and Guide Visual-Grounded Reasoning in Diffusion Multimodal Language Models
- Lookahead Unmasking Elicits Accurate Decoding in Diffusion Language Models
- A Comprehensive Study on Visual Token Redundancy for Discrete Diffusion-based Multimodal Large Language Models
- Diffusion Language Models Know the Answer Before Decoding
- Lavida-O: Elastic Large Masked Diffusion Models for Unified Multimodal Understanding and Generation
- Sparse-LaViDa: Sparse Multimodal Discrete Diffusion Language Models
- DocVQA: A Dataset for VQA on Document Images
- Muddit: Liberating Generation Beyond Text-to-Image with a Unified Discrete Diffusion Model
- Accelerating Diffusion Large Language Models with SlowFast Sampling: The Three Golden Principles
- Fast-dLLM: Training-free Acceleration of Diffusion LLM by Enabling KV Cache and Parallel Decoding
- RedVTP: Training-Free Acceleration of Diffusion Vision-Language Models Inference via Masked Token-Guided Visual Token Pruning
- Improving Sampling for Masked Diffusion Models via Information Gain
- Dream 7B: Diffusion Large Language Models
- Dimple: Discrete Diffusion Multimodal Large Language Model with Parallel Decoding
Related papers
- Loss Knows Best: Detecting Annotation Errors in Videos via Loss Trajectories
- AnchorWeave: World-Consistent Video Generation with Retrieved Local Spatial Memories
- Benchmarking the Robustness of Foundation Models for Mammography under Domain Shift
- MambaX-Net: Dual-Input Mamba-Enhanced Cross-Attention Network for Longitudinal MRI Segmentation
- TeleOCR: Navigating Document Parsing Across Digital and Camera-Captured Documents
- A Survey on Efficient Vision-Language-Action Models