Beyond Attention Imbalance: Mitigating Hallucinations via Spectral Surgery
cs.CV
Submitted: 2026-09-29
Updated: 2026-09-29
Code: https://github.com/a6103121/FLASH
License: http://creativecommons.org/licenses/by/4.0/
The gist: While Large Vision-Language Models (LVLMs) achieve remarkable success, hallucinations remain a significant barrier to their reliable deployment.
Terminology
Abstract
While Large Vision-Language Models (LVLMs) achieve remarkable success, hallucinations remain a significant barrier to their reliable deployment. Recent studies primarily attribute these issues to cross-modal attention imbalances; most solutions therefore focus on reweighting visual tokens or suppressing language priors. However, such approaches often overlook the spectral characteristics of the visual information flow and frequently rely on Contrastive Decoding (CD), which doubles inference time. Instead of following conventional approaches, we identify two distinct hallucination patterns-Perceptual-Semantic Dissociation and Localized Fixation-and propose FLASH (Frequency-Localized Attention SHaping), a training-free and CD-free framework. FLASH utilizes a Spectral Vortex Score to detect vision heads within multi-head attention layers and applies adaptive spectral modulation to rectify the visual information flow during decoding. Empirical results demonstrate that FLASH achieves a superior balance between performance and efficiency compared to SOTA methods.
Sources
- ShareGPT4V: Improving Large Multi-Modal Models with Better Captions
- Why Is Spatial Reasoning Hard for VLMs? An Attention Mechanism Perspective on Focus Areas
- Do You Keep an Eye on What I Ask? Mitigating Multimodal Hallucination via Attention-Guided Ensemble Decoding
- DoLa: Decoding by Contrasting Layers Improves Factuality in Large Language Models
- MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Models
- Detecting and Preventing Hallucinations in Large Vision Language Models
- OPERA: Alleviating Hallucination in Multi-Modal Large Language Models via Over-Trust Penalty and Retrospection-Allocation
- GQA: A New Dataset for Real-World Visual Reasoning and Compositional Question Answering
- Self-Introspective Decoding: Alleviating Hallucinations for Large Vision-Language Models
- Silkie: Preference Distillation for Large Visual Language Models
- Seeing but Not Believing: Probing the Disconnect Between Visual Attention and Answer Correctness in VLMs
- Contrastive Decoding Improves Reasoning in Large Language Models
- How Do Vision Transformers Work?
- Direct Preference Optimization: Your Language Model is Secretly a Reward Model
- A-OKVQA: A Benchmark for Visual Question Answering using World Knowledge
- Aligning Large Multimodal Models with Factually Augmented RLHF
- mDPO: Conditional Preference Optimization for Multimodal Large Language Models
- High Frequency Component Helps Explain the Generalization of Convolutional Neural Networks
- AMBER: An LLM-free Multi-dimensional Benchmark for MLLMs Hallucination Evaluation
- Mitigating Hallucinations in Large Vision-Language Models with Instruction Contrastive Decoding
Related papers
- Loss Knows Best: Detecting Annotation Errors in Videos via Loss Trajectories
- AnchorWeave: World-Consistent Video Generation with Retrieved Local Spatial Memories
- Benchmarking the Robustness of Foundation Models for Mammography under Domain Shift
- MambaX-Net: Dual-Input Mamba-Enhanced Cross-Attention Network for Longitudinal MRI Segmentation
- TeleOCR: Navigating Document Parsing Across Digital and Camera-Captured Documents
- A Survey on Efficient Vision-Language-Action Models