Wiener Representation Filtering for VLM Hallucination Suppression
Ameen Ali, Tamim Zoabi, Lidor Brami, Lior Wolf
cs.CV, cs.LG
Submitted: 2026-08-08
Updated: 2026-08-11
License: http://creativecommons.org/licenses/by/4.0/
The gist: Vision-language models (VLMs) excel at open-ended captioning and visual QA but often describe objects, attributes, or relations absent from the image, a phenomenon known as object hallucination.
Terminology
Abstract
Vision-language models (VLMs) excel at open-ended captioning and visual QA but often describe objects, attributes, or relations absent from the image, a phenomenon known as object hallucination. We propose a training-free, post-hoc representation editing technique that operates in the representation space of the language backbone. The method performs a lightweight, one-time offline calibration on a modest paired dataset to estimate the required covariance structures, using only forward passes and empirical second-order statistics with no gradient updates or fine-tuning, after which the correction is absorbed directly into the model's existing weights. By modeling hidden states as a superposition of truthful and hallucination-associated components, we derive a Wiener-type estimator whose optimal gains are given in closed form from the covariances of paired truthful and hallucinated representations. An eigendecomposition yields mode-wise attenuation that respects a stability criterion, i.e., the filter responds continuously to estimation noise. The correction is applied once to the feed-forward output projections of selected deeper layers, at inference time, the model runs unchanged and at the same speed. Experiments on LLaVA-1.5, MiniGPT-4, Gemma3, and mPLUG-Owl2 demonstrate consistent reductions in object hallucination on CHAIR, POPE, and MME while maintaining caption fluency and overall response quality. We further demonstrate the generality of our approach on the TempCompass video understanding benchmark and on discrete diffusion language models for grounded dialogue, showing that representation filtering reduces hallucinations even in temporal video reasoning and multi-step, sequence-wide denoising settings.
Sources
- Gaps or Hallucinations? Gazing into Machine-Generated Legal Analysis for Fine-grained Text Evaluations
- Hallucination of Multimodal Large Language Models: A Survey
- MiniGPT-v2: large language model as a unified interface for vision-language multi-task learning
- DoLa: Decoding by Contrasting Layers Improves Factuality in Large Language Models
- FaithDial: A Faithful Benchmark for Information-Seeking Dialogue
- MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Models
- FGAIF: Aligning Large Vision-Language Models with Fine-grained AI Feedback
- HaluEval: A Large-Scale Hallucination Evaluation Benchmark for Large Language Models
- TruthfulQA: Measuring How Models Mimic Human Falsehoods
- A Survey on Hallucination in Large Vision-Language Models
- TempCompass: Do Video LLMs Really Understand Videos?
- LLaMA: Open and Efficient Foundation Language Models
- Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolution
- Analyzing and Mitigating Object Hallucination in Large Vision-Language Models
- MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Models
Related papers
- Loss Knows Best: Detecting Annotation Errors in Videos via Loss Trajectories
- AnchorWeave: World-Consistent Video Generation with Retrieved Local Spatial Memories
- Benchmarking the Robustness of Foundation Models for Mammography under Domain Shift
- MambaX-Net: Dual-Input Mamba-Enhanced Cross-Attention Network for Longitudinal MRI Segmentation
- TeleOCR: Navigating Document Parsing Across Digital and Camera-Captured Documents
- A Survey on Efficient Vision-Language-Action Models