Readout is not Recovery: Dissociating Coordinate Emission from Visual-Corruption Repair in Vision-Language Models
cs.AI, cs.CV, cs.LG
Submitted: 2026-09-26
Updated: 2026-09-26
Comments: 29 pages (14 main + appendix), 2 figures, 13 tables
License: http://creativecommons.org/licenses/by/4.0/
The gist: VLM bounding-box localization is both language generation and spatial commitment.
Terminology
Abstract
VLM bounding-box localization is both language generation and spatial commitment. Parseable fields such as bbox 2d make localization easy to score, but dimensions that emit coordinate tokens need not repair localization after visual evidence is damaged. We study this readout/recovery separation in Qwen3-VL-4B-Instruct on single-object COCO grounding. We compare clean coordinate-token readout rankings with corruption-derived repair rankings, using object-mask endpoint replacement for recovery and clean-input flooring for depth localization. In Qwen3-VL, coordinate-token rankings are inert through layer 24, load-bearing from layers 32-35, and peak at layer 34; corruption-derived rankings harm layers 16-24 but become beneficial near layer 35/final. A Kimi-VL-A3B diagnostic shows a matching output-proximal transition despite a different box format. Object-mask recovery separates rank budgets: k=250 shows necessity, k=500 shows Top- k restoration above random, and k=d/2 is largely capacity-driven. Partial-occlusion sweeps reveal that high-overlap coordinate-token sets can hurt at k=1000 and help mainly at half-width, while population corruption-derived sets provide no reliable fixed repair set. Edge-attribution patching shows coordinate-token paths are high precision but low recall for detection recovery, and RMSNorm quasi-layer controls do not close the endpoint-repair gap. Endpoint coordinate triage is therefore a useful circuit prior, but occlusion recovery requires a separate benchmark.
Sources
- Layer Normalization
- Qwen3-VL Technical Report
- Shikra: Unleashing Multimodal LLM's Referential Dialogue Magic
- Kimi-VL Technical Report
- Benchmarking Robustness in Object Detection: Autonomous Driving when Winter is Coming
- Kosmos-2: Grounding Multimodal Large Language Models to the World
Related papers
- MAVEN-T: Reinforced Heterogeneous Distillation for Real-Time Multi-Agent Trajectory Prediction
- Model Discovery Agent: LLM-assisted Bayesian experiment design for data-efficient discovery of mechanistic world models
- Agents' Last Exam
- The Clinician's Veto: Navigating Trust, Liability, and Uncertainty in Autonomous AI Prescribing
- MindHelper: Closed-Loop Embodied Mental-State Reasoning for Precision Intervention
- Incumbent Advantage: Brand Bias and Cognitive Manipulation Dynamics in LLM Recommendation Systems