VisER: Visual Evidence and Reliance for Object Hallucination Detection in LVLMs

arXiv:2608.30480 · cs.CV, cs.LG · Submitted 2026-08-31 · Read on arXiv

cs.CV, cs.LG

Submitted: 2026-08-31

Updated: 2026-08-31

Comments: Accepted at EMNLP 2026 Main Conference

Code: https://github.com/AfsanehEB/VisER

License: http://creativecommons.org/licenses/by/4.0/

The gist: Object hallucination remains a persistent reliability issue in large vision-language models, where generated object mentions may sound plausible but lack visual grounding.

Terminology

Abstract

Object hallucination remains a persistent reliability issue in large vision-language models, where generated object mentions may sound plausible but lack visual grounding. Recent training-free detectors use internal signals such as token likelihood, attention, visual confidence, or image-text similarity to identify hallucinated objects. These signals are useful, but they are often source-confounded. They measure how strongly an object is supported inside the model without distinguishing whether that support comes from object-specific visual evidence or the generated text prefix. In difficult cases, a hallucinated object can still receive high internal support because it fits the scene, is associated with nearby visual cues, or follows naturally from the generated text prefix. We propose VisER, a training-free two-sided metric for object-level hallucination detection. VisER evaluates each generated object mention from two complementary views. Visual Evidence measures whether object-context compatibility is backed by object-specific evidence from image tokens. Visual Reliance measures whether the object is supported more by the image than by the generated prefix. Combining these views gives a more source-aware grounding score, while avoiding additional object-level verification generations. Across multiple LVLMs and benchmarks, VisER improves AUROC and AUPR over a range of baselines.

Sources

Related papers