What Keeps Vision-Language Models Looking at the Image?

arXiv:2607.12815 · cs.AI · Submitted 2026-07-14 · Read on arXiv

cs.AI

Submitted: 2026-07-14

Updated: 2026-10-04

Terminology

• Trace — 2× in this paper · explained in 2 episodes
• Vision-language models — 2× in this paper · explained in 2 episodes
• Accuracy — 1× in this paper · explained in 1 episode
• Chain-of-Thought (CoT) — 1× in this paper · explained in 4 episodes
• Chain-of-Thought (CoT) Prompting — 1× in this paper · explained in 3 episodes
• Decode — 1× in this paper · explained in 2 episodes
• Reasoning Traces — 1× in this paper · explained in 3 episodes
• Test-time Scaling — 1× in this paper · explained in 2 episodes
• Vision-Language Models (VLMs) — 1× in this paper · explained in 6 episodes
• Visual Access Boundary — 1× in this paper

Sources

• Ask Twice, Look Twice: Prompt Echoing Resolves the Question-First Paradox in Vision-Language Models
• TallyQA: Answering Complex Counting Questions
• Qwen2.5-VL Technical Report
• Seen, Said, or Forgotten? A Causal Audit of Visual KV Memory Across Dialog Turns
• An Image is Worth 1/2 Tokens After Layer 2: Plug-and-Play Inference Acceleration for Large Vision-Language Models
• Qwen Look Again: Guiding Vision-Language Reasoning Models to Re-attention Visual Information
• Hidden in plain sight: VLMs overlook their visual representations
• How to use and interpret activation patching
• GQA: A New Dataset for Real-World Visual Reasoning and Compositional Question Answering
• Towards Faithfully Interpretable NLP Systems: How should we define and evaluate faithfulness?
• Attention is not Explanation
• CLEVR: A Diagnostic Dataset for Compositional Language and Elementary Visual Reasoning
• What's in the Image? A Deep-Dive into the Vision of Vision Language Models
• SparseVILA: Decoupling Visual Sparsity for Efficient VLM Inference
• Large Language Models are Zero-Shot Reasoners
• Measuring Faithfulness in Chain-of-Thought Reasoning
• Visual Representations inside the Language Model
• Show Your Work: Scratchpads for Intermediate Computation with Language Models
• Let's Think Dot by Dot: Hidden Computation in Transformer Language Models
• VLMs Need Words: Vision Language Models Ignore Visual Detail In Favor of Semantic Anchors

Related papers

• MAVEN-T: Reinforced Heterogeneous Distillation for Real-Time Multi-Agent Trajectory Prediction
• Model Discovery Agent: LLM-assisted Bayesian experiment design for data-efficient discovery of mechanistic world models
• Agents' Last Exam
• The Clinician's Veto: Navigating Trust, Liability, and Uncertainty in Autonomous AI Prescribing
• MindHelper: Closed-Loop Embodied Mental-State Reasoning for Precision Intervention
• Incumbent Advantage: Brand Bias and Cognitive Manipulation Dynamics in LLM Recommendation Systems