Mitigating Hallucination in Large Vision-Language Models via Adaptive Attention Calibration

arXiv:2505.21472 · cs.CV, cs.CL, cs.MM · Submitted 2025-05-27 · Read on arXiv

cs.CV, cs.CL, cs.MM

Submitted: 2025-05-27

Updated: 2025-08-11

Code: https://github.com/mehrdadfazli/CAAC

Terminology

• Calibration — 3× in this paper · explained in 7 episodes
• Hallucination — 3× in this paper · explained in 4 episodes
• Vision-language models — 2× in this paper · explained in 2 episodes
• Accuracy — 1× in this paper · explained in 1 episode
• Alignment — 1× in this paper · explained in 1 episode
• Benchmark — 1× in this paper · explained in 2 episodes
• Confidence-Aware Attention Calibration — 1× in this paper
• Grounding — 1× in this paper · explained in 3 episodes
• Visual Grounding — 1× in this paper · explained in 2 episodes

Sources

• Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyond
• Hallucination of Multimodal Large Language Models: A Survey
• Shikra: Unleashing Multimodal LLM's Referential Dialogue Magic
• InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Tasks
• InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning
• HallusionBench: An Advanced Diagnostic Suite for Entangled Language Hallucination and Visual Illusion in Large Vision-Language Models
• Contrastive Decoding: Open-ended Text Generation as Optimization
• Evaluating Object Hallucination in Large Vision-Language Models
• Microsoft COCO: Common Objects in Context
• Mitigating Hallucination in Large Multi-Modal Models via Robust Instruction Tuning
• Visual Instruction Tuning
• A Survey on Hallucination in Large Vision-Language Models
• Paying More Attention to Image: A Training-Free Method for Alleviating Hallucination in LVLMs
• Learning Transferable Visual Models From Natural Language Supervision
• Object Hallucination in Image Captioning
• A Thorough Examination of Decoding Methods in the Era of LLMs
• Octopus: Alleviating Hallucination via Dynamic Contrastive Decoding
• LLaMA: Open and Efficient Foundation Language Models
• AMBER: An LLM-free Multi-dimensional Benchmark for MLLMs Hallucination Evaluation
• Don't Miss the Forest for the Trees: Attentional Vision Calibration for Large Vision Language Models

Related papers

• Loss Knows Best: Detecting Annotation Errors in Videos via Loss Trajectories
• AnchorWeave: World-Consistent Video Generation with Retrieved Local Spatial Memories
• Benchmarking the Robustness of Foundation Models for Mammography under Domain Shift
• MambaX-Net: Dual-Input Mamba-Enhanced Cross-Attention Network for Longitudinal MRI Segmentation
• TeleOCR: Navigating Document Parsing Across Digital and Camera-Captured Documents
• A Survey on Efficient Vision-Language-Action Models