Revealing Multi-View Hallucination in Large Vision-Language Models
cs.CV, cs.AI
Submitted: 2026-03-25
Updated: 2026-09-01
Comments: Accepted to EMNLP 2026 (Main)
License: http://creativecommons.org/licenses/by/4.0/
The gist: Large vision-language models (LVLMs) are increasingly being applied to multi-view image inputs captured from diverse viewpoints.
Terminology
Abstract
Large vision-language models (LVLMs) are increasingly being applied to multi-view image inputs captured from diverse viewpoints. Despite this growing use, current LVLMs often generate incorrect responses due to visual interference from non-target instances or viewpoints, a phenomenon we term multi-view hallucination (MVH). To systematically analyze this problem, we construct MVH-Bench, a benchmark comprising 4.8k question-answer pairs targeting two types of hallucination: cross-instance and cross-view. Empirical results show that MVH is prevalent across recent LVLMs. To address this issue, we propose Reference Shift Contrastive Decoding (RSCD), a training-free decoding technique that suppresses visual interference by generating negative logits through attention masking. Experiments on MVH-Bench with LLaVA-OneVision and Qwen2.5-VL demonstrate that RSCD improves performance by up to 25.7 and 94.8 points over existing hallucination mitigation methods, highlighting the effectiveness of RSCD.
Sources
- Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilities
- Seeing is Believing: Mitigating Hallucination in Large Vision-Language Models via CLIP-Guided Decoding
- Exploring Hallucination of Large Multimodal Models in Video Understanding: Benchmark, Analysis and Mitigation
- Qwen2.5-VL Technical Report
- DAMRO: Dive into the Attention Mechanism of LVLM to Reduce Object Hallucination
- An Introduction to Vision-Language Modeling
- Matterport3D: Learning from RGB-D Data in Indoor Environments
- VidHal: Benchmarking Temporal Hallucinations in Vision LLMs
- LLaVA-OneVision: Easy Visual Task Transfer
- GPT-4o System Card
- LLaVA-NeXT-Interleave: Tackling Multi-image, Video, and 3D in Large Multimodal Models
- Evaluating Object Hallucination in Large Vision-Language Models
- A Survey of State of the Art Large Vision Language Models: Alignment, Benchmark, Evaluations and Challenges
- VACoDe: Visual Augmented Contrastive Decoding
- DeepSeek-VL: Towards Real-World Vision-Language Understanding
- Towards Comprehensive Scene Understanding: Integrating First and Third-Person Views for LVLMs
- Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolution
- InternVL3.5: Advancing Open-Source Multimodal Models in Versatility, Reasoning, and Efficiency
- Mitigating Hallucinations in Large Vision-Language Models with Instruction Contrastive Decoding
- UniDepthV2: Universal Monocular Metric Depth Estimation Made Simpler
Related papers
- Loss Knows Best: Detecting Annotation Errors in Videos via Loss Trajectories
- AnchorWeave: World-Consistent Video Generation with Retrieved Local Spatial Memories
- Benchmarking the Robustness of Foundation Models for Mammography under Domain Shift
- MambaX-Net: Dual-Input Mamba-Enhanced Cross-Attention Network for Longitudinal MRI Segmentation
- TeleOCR: Navigating Document Parsing Across Digital and Camera-Captured Documents
- A Survey on Efficient Vision-Language-Action Models