Vision Is Not Overhead: One-Pass Block Drafting for Lossless Speculative Decoding in Vision-Language Models
cs.AI, cs.CL, cs.CV
Submitted: 2026-08-31
Updated: 2026-10-06
Comments: 21 pages, 8 figures, 16 tables. Code: https://github.com/js-lee-AI/GLANCE
Code: https://github.com/js-lee-AI/GLANCE
License: http://creativecommons.org/licenses/by/4.0/
The gist: Speculative decoding accelerates generation without changing its output, but on vision-language models (VLMs) a self-reinforcing cycle holds it back.
Terminology
Abstract
Speculative decoding accelerates generation without changing its output, but on vision-language models (VLMs) a self-reinforcing cycle holds it back. Because an autoregressive drafter pays a sequential pass for each drafted token, it must stay small and can ill afford to attend to the image at each pass. Prior work therefore compresses or hides the image, leaving the drafter weakest on the text the image determines. We present GLANCE, a one-pass block drafter that breaks this cycle on an unmodified VLM target. Its block-diffusion head drafts a whole block in one forward pass over the target's already fused vision-language states, reading the multimodal context once, however deep the draft. The target verifies a wide candidate tree in one pass and commits exactly its greedy output. In one production engine at a fixed round budget, GLANCE decodes up to 3.05 times faster than autoregressive decoding and outpaces the production EAGLE3-VL head on average and by about 11% on grounded tasks. An entropy law explains when drafting pays, predicting the longest accepted blocks on grounded tasks, where the target's next-token entropy is lowest. Our code is available at https://github.com/js-lee-AI/GLANCE.
Sources
- AdaEDL: Early Draft Stopping for Speculative Decoding of Large Language Models via an Entropy-based Lower Bound on Token Acceptance Probability
- Hydra: Sequentially-Dependent Draft Heads for Medusa Decoding
- Block Diffusion: Interpolating Between Autoregressive and Diffusion Language Models
- Qwen2.5-VL Technical Report
- Accelerating Large Language Model Decoding with Speculative Sampling
- ALLaVA: Harnessing GPT4V-Synthesized Data for Lite Vision-Language Models
- DFlash: Block Diffusion for Flash Speculative Decoding
- Sequoia: Scalable, Robust, and Hardware-aware Speculative Decoding
- DEER: Draft with Diffusion, Verify with Autoregressive Models
- Speculative Diffusion Decoding: Accelerating Language Generation through Diffusion
- Break the Sequential Dependency of LLM Inference Using Lookahead Decoding
- On Speculative Decoding for Multimodal Large Language Models
- MASSV: Multimodal Adaptation and Self-Data Distillation for Speculative Decoding of Vision-Language Models
- DREAM: Drafting with Refined Target Features and Entropy-Adaptive Cross-Attention Fusion for Multimodal Speculative Decoding
- SpecVLM: Fast Speculative Decoding in Vision-Language Models
- SpecDec++: Boosting Speculative Decoding via Adaptive Candidate Lengths
- ViSpec: Accelerating Vision-Language Models with Vision-Aware Speculative Decoding
- DiffuSpec: Unlocking Diffusion Language Models for Speculative Decoding
- EAGLE-3: Scaling up Inference Acceleration of Large Language Models via Training-Time Test
- Acceptance Dynamics Across Cognitive Domains in Speculative Decoding
Related papers
- MAVEN-T: Reinforced Heterogeneous Distillation for Real-Time Multi-Agent Trajectory Prediction
- Model Discovery Agent: LLM-assisted Bayesian experiment design for data-efficient discovery of mechanistic world models
- Agents' Last Exam
- The Clinician's Veto: Navigating Trust, Liability, and Uncertainty in Autonomous AI Prescribing
- MindHelper: Closed-Loop Embodied Mental-State Reasoning for Precision Intervention
- Incumbent Advantage: Brand Bias and Cognitive Manipulation Dynamics in LLM Recommendation Systems