DARE to Mitigate Hallucination: Dual-path Auto-Regressive-aware Editing
cs.CV
Submitted: 2026-09-29
Updated: 2026-09-29
Comments: 19 pages, Accepted at ECCV 2026
Code: https://github.com/KU-VGI/DARE
License: http://creativecommons.org/licenses/by/4.0/
The gist: Large vision-language models (LVLMs) have recently achieved remarkable progress across multimodal tasks, yet object hallucination remains a persistent challenge where models generate descriptions
Terminology
Abstract
Large vision-language models (LVLMs) have recently achieved remarkable progress across multimodal tasks, yet object hallucination remains a persistent challenge where models generate descriptions inconsistent with the visual input. Recent work mitigates hallucinations through training-free representation editing, typically by constructing hallucination-related directions from teacher-forcing (TF) contrasts between hallucinated and truthful responses. However, LVLMs operate through autoregressive (AR) decoding during generation, raising the question of whether TF-based analysis fully reflects the generation dynamics that lead to hallucinated outputs. In this paper, we analyze the relationship between TF-based editing and AR generation behavior and find that TF-based editing alone may be insufficient to capture both decoding dynamics and multimodal interactions associated with hallucinations. To address this limitation, we propose DARE (Dual-path Auto-Regressive-aware Editing), a hybrid hallucination editing framework that integrates two complementary contrast pathways: textual contrasts and image contrasts, together with autoregressive-aware representation signals. Specifically, DARE constructs hallucination editing directions from (1) TF-based textual contrasts, (2) AR-aware representation transitions during decoding, and (3) controlled visual differences between paired images. Extensive experiments on multiple LVLM hallucination benchmarks demonstrate that DARE consistently reduces object hallucinations while preserving multimodal perception capability and inference efficiency. Our implementation code is available at https://github.com/KU-VGI/DARE.
Sources
- Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyond
- Qwen3-VL Technical Report
- Shikra: Unleashing Multimodal LLM's Referential Dialogue Magic
- HALC: Object Hallucination Reduction via Adaptive Focal-Contrast Decoding
- Beam Search Strategies for Neural Machine Translation
- MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Models
- Contrastive Decoding: Open-ended Text Generation as Optimization
- Mitigating Hallucination in Large Multi-Modal Models via Robust Instruction Tuning
- Improved Baselines with Visual Instruction Tuning
- Investigating and Mitigating Object Hallucinations in Pretrained Vision-Language (CLIP) Models
- Instruction Tuning with GPT-4
- A Survey of Hallucination in Large Foundation Models
- Object Hallucination in Image Captioning
- Gemma 3 Technical Report
- Beyond Hallucinations: Enhancing LVLMs through Hallucination-Aware Direct Preference Optimization
- Analyzing and Mitigating Object Hallucination in Large Vision-Language Models
- MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Models
Related papers
- Loss Knows Best: Detecting Annotation Errors in Videos via Loss Trajectories
- AnchorWeave: World-Consistent Video Generation with Retrieved Local Spatial Memories
- Benchmarking the Robustness of Foundation Models for Mammography under Domain Shift
- MambaX-Net: Dual-Input Mamba-Enhanced Cross-Attention Network for Longitudinal MRI Segmentation
- TeleOCR: Navigating Document Parsing Across Digital and Camera-Captured Documents
- A Survey on Efficient Vision-Language-Action Models