Balancing Efficiency and Efficacy: Training-Free Attention-Guided Switching Between Explicit and Latent Thoughts for MLLMs
Haoqian Kang, Liupeng Li, Kuofeng Gao, Jinpeng Wang, Zhenyu Lu, Bin Chen, Ke Chen, Yaowei Wang
cs.MM, cs.AI, cs.CL, cs.CV
Submitted: 2026-08-04
Comments: Accepted by ACM MM 2026. 10 pages, 6 figures, 5 tables
Code: https://github.com/swordAndSnow/MM26-AGS
License: http://creativecommons.org/licenses/by/4.0/
The gist: Reasoning in Multimodal Large Language Models (MLLMs) requires both fine-grained visual perception and rigorous logical deduction.
Terminology
Abstract
Reasoning in Multimodal Large Language Models (MLLMs) requires both fine-grained visual perception and rigorous logical deduction. Explicit text-based Chain-of-Thought (CoT) is computationally expensive and prone to visual hallucinations, while existing latent reasoning methods typically require costly training. Furthermore, directly adapting training-free LLM reasoning mechanisms to the multimodal setting yields unstable performance. We identify that this failure stems from their reliance on token-level entropy, which fundamentally conflates perceptual ambiguity (e.g., unclear visual details) with logical uncertainty (e.g., complex reasoning steps). To overcome this bottleneck, we present a novel training-free inference strategy for MLLMs that explicitly decouples perception and reasoning. We propose a novel metric, the vision-to-text attention ratio, to dynamically gauge the model's cognitive focus. Guided by this metric, our proposed framework, Attention-Guided Switching (AGS), adaptively triggers latent reasoning for perceptual tokens to preserve high-fidelity visual information in the continuous space, while enforcing explicit text generation for logical tokens to maintain structural anchoring. Extensive experiments demonstrate that our method achieves state-of-the-art performance, significantly improving both accuracy and inference efficiency by reducing autoregressive steps and latency. Code is released at https://github.com/swordAndSnow/MM26-AGS.
Sources
- Qwen3-VL Technical Report
- SFT or RL? An Early Investigation into Training R1-Like Reasoning Large Vision-Language Models
- Visual Thoughts: A Unified Perspective of Understanding Multimodal Chain-of-Thought
- SophiaVL-R1: Reinforcing MLLMs Reasoning with Thinking Reward
- Reasoning-Aligned Perception Decoupling for Scalable Multi-modal Reasoning
- DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning
- Training Large Language Models to Reason in a Continuous Latent Space
- Spotlight on Token Perception for Multimodal Reinforcement Learning
- Vision-R1: Incentivizing Reasoning Capability in Multimodal Large Language Models
- Mitigating Visual Context Degradation in Large Multimodal Models: A Training-Free Decoupled Agentic Framework
- Mini-o3: Scaling Up Reasoning Patterns and Interaction Turns for Visual Search
- Latent Visual Reasoning
- LLaVA-OneVision: Easy Visual Task Transfer
- CVSearch: Empowering Multimodal LLMs with Cognitive Visual Search for High-Resolution Image Perception
- Rethinking Token-Level Policy Optimization for Multimodal Chain-of-Thought
- Bridging Perception and Reasoning: Token Reweighting for RLVR in Multimodal LLMs
- MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contexts
- SegCompass: Exploring Interpretable Alignment with Sparse Autoencoders for Enhanced Reasoning Segmentation
- MM-Eureka: Exploring the Frontiers of Multimodal Reasoning with Rule-based Reinforcement Learning
- Multimodal Chain of Continuous Thought for Latent-Space Reasoning in Vision-Language Models
Related papers
- A Survey of Generative Categories and Techniques in Multimodal Generative Models
- ControlFoley: Unified and Controllable Video-to-Audio Generation with Cross-Modal Conflict Handling
- EyeNavGS: A 6-DoF Navigation Dataset and Record-n-Replay Software for Real-World 3DGS Scenes in VR
- Mechanisms of Multimodal Synchronization: Insights from Decoder-Based Video-Text-to-Speech Synthesis
- Zero-shot Video Moment Retrieval via Off-the-shelf Multimodal Large Language Models
- Mitigating GenAI-powered Evidence Pollution for Out-of-Context Multimodal Misinformation Detection