When Helpful Text Hurts: Option-Redirecting Bias in Vision-Language Models
cs.AI, cs.CV
Submitted: 2026-09-26
Updated: 2026-09-26
Comments: Accepted at ACM Multimedia 2026 (ACM MM 2026). 25 pages, 16 figures. This arXiv version includes supplementary material
License: http://creativecommons.org/licenses/by/4.0/
The gist: In tri-modal visual question answering (VQA), auxiliary text is commonly used to complement visual and textual inputs, yet its reliability is often uncontrolled.
Terminology
Abstract
In tri-modal visual question answering (VQA), auxiliary text is commonly used to complement visual and textual inputs, yet its reliability is often uncontrolled. While prior work studies modality conflicts in general, it remains unclear how different types of unreliable auxiliary text affect answer selection under fixed image-question-option contexts. In this work, we show that the most harmful auxiliary text is not necessarily the most factually incorrect, but the one that aligns with the question while contradicting the image and favoring a specific distractor, leading to systematic redirection of model predictions. To isolate this effect, we introduce the Textual Reliability Ladder, a controlled diagnostic protocol that decomposes auxiliary text along three axes: image consistency, question relevance, and option support. Across multiple datasets (ScienceQA, VCR, A-OKVQA, Causal-VidQA) and recent VLMs, we find that such distractor-supporting text induces the largest accuracy drops (up to 53.1%) and concentrates errors on specific incorrect options. To mitigate this failure mode, we propose a training-free inference-time intervention that explicitly counteracts this redirection effect via noise-stability steering and dynamic grounding, reducing redirected errors while largely preserving performance under faithful text. Our results highlight that auxiliary-text reliability must be understood at the decision level, rather than solely through factual correctness, and provide a practical pathway toward more robust tri-modal reasoning.
Sources
- Qwen3-VL Technical Report
- Qwen2.5-VL Technical Report
- Hallucination of Multimodal Large Language Models: A Survey
- Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilities
- OCRBench v2: An Improved Benchmark for Evaluating Large Multimodal Models on Visual Text Localization and Reasoning
- Hidden in plain sight: VLMs overlook their visual representations
- Textual Steering Vectors Can Improve Visual Understanding in Multimodal Large Language Models
- How Do Vision-Language Models Process Conflicting Information Across Modalities?
- Revisiting the Role of Language Priors in Vision-Language Models
- A Survey on Hallucination in Large Vision-Language Models
- Insight Over Sight: Exploring the Vision-Knowledge Conflicts in Multimodal LLMs
- Pointer Sentinel Mixture Models
- Challenges in Understanding Modality Conflict in Vision-Language Models
- Steer LLM Latents for Hallucination Detection
- Web Artifact Attacks Disrupt Vision Language Models
- CrossCheck-Bench: Diagnosing Compositional Failures in Multimodal Conflict Resolution
- InternVL3.5: Advancing Open-Source Multimodal Models in Versatility, Reasoning, and Efficiency
- C-TPT: Calibrated Test-Time Prompt Tuning for Vision-Language Models via Text Feature Dispersion
- Mitigating Easy Option Bias in Multiple-Choice Question Answering
- Robust Multimodal Large Language Models Against Modality Conflict
Related papers
- MAVEN-T: Reinforced Heterogeneous Distillation for Real-Time Multi-Agent Trajectory Prediction
- Model Discovery Agent: LLM-assisted Bayesian experiment design for data-efficient discovery of mechanistic world models
- Agents' Last Exam
- The Clinician's Veto: Navigating Trust, Liability, and Uncertainty in Autonomous AI Prescribing
- MindHelper: Closed-Loop Embodied Mental-State Reasoning for Precision Intervention
- Incumbent Advantage: Brand Bias and Cognitive Manipulation Dynamics in LLM Recommendation Systems