Same Attention, Different Truths: Put Logit-Lens over Visual Attention to Detect and Mitigate LVLM Object Hallucination

The paper addresses object hallucination in Large Vision-Language Models (LVLMs)—the phenomenon where models generate objects that are absent from the image. The authors challenge the prevailing view that attributes hallucination to insufficient visual attention. Prior work argued that textual priors dominate the multimodal interaction and suppress visual focus or that improper processing of visual information leads to inadequate attention to salient regions, leading to methods that attempt to amplify or redistribute attention or inject extra visual guidance during training/inference.

The authors state their central claim directly: our experiments reveal that this explanation is incomplete: the key issue may not be how much the model attends to the image, but what it attends to and why. This is encapsulated in the paper's title phrase, Same Attention, Different Truths.

Using LLaVA-1.5-7B on 500 randomly sampled COCO 2014 images, the authors analyze attention distributions of generated tokens across layers. They identify a four-stage attention pattern for object tokens:

• Initialization Stage (layers 1–2): high attention to both image and text

• Text-Attention Stage (layers 3–19): text attention dominates

• Image-Attention Stage (layers 20–27): attention to the image increases and becomes dominant, while text attention drops. The model focuses on extracting evidence related to the candidate objects

• Language-Organization Stage (layers 28–32): image attention decreases, text attention rises

Critically, they find that real and hallucinated object tokens show no systematic difference in attention magnitude during the Image-Attention Stage. Using CHAIR to label real vs. hallucinated objects, they quantify average image attention and find no systematic difference between the two categories. Qualitatively, both real and hallucinated object tokens focus their attention on specific regions during the Image-Attention Stage, with comparable magnitudes. The authors conclude: object hallucination is not caused by insufficient visual attention.

The authors apply the Logit Lens—an interpretability method that decodes the hidden states of intermediate layers into vocabulary space—to the hidden states of top-k attended image tokens during the Image-Attention Stage. Their findings:

• For Real Objects: "the highly-attended visual regions (i.e., the actual suitcase regions) can also be decoded to consistent tokens like 'suit' or 'lug' by Logit-Lens. This indicates that the real token attention is faithfully grounded in the visual evidence."

• For Hallucinated Objects: a clear semantic inconsistency emerges. The highly-attended visual regions (e.g., a 'blurry patch on the ground') do not decode to the hallucinated target token by Logit-Lens.

This reveals that hallucinated object tokens exhibit a semantic inconsistency, where the visual evidence extracted by the attention mechanism does not support the generated token.

Through a High-Attention Regions Masking intervention experiment, the authors identify two distinct hallucination types:

When masking the high-attention regions, the hallucinated token disappears, indicating that the hallucination is visually rooted. These regions are often ambiguous, blurry, or uncertain areas (e.g., a round, ceramic-like area leading to the generation of 'bowl'; or a dark, blurry rectangular region resulting in the output 'cell phone'). The model may attempt to forcibly extract certain semantics from highly uncertain visual evidence, which is insufficient to support the generated object.

In contrast, masking the high-attention regions does not remove the hallucinated token. Instead, hallucination persists and attention shifts to other regions. Logit Lens analysis shows these attended regions can be decoded into the correct objects actually present in the image, yet the model still outputs the hallucinated token. This is consistent with the generation being dominated by strong contextual priors (e.g., outputting 'microwave' in a kitchen description even when no microwave is present). Here, "the attention mechanism is more like a procedural 'routine': its internal rules require it to 'attend somewhere' before generating an object. So it arbitrarily 'grounds' its attention on some regions to satisfy this procedural requirement, while the actual generation decision is mainly controlled by the contextual prior."

Statistical analysis on LLaVA-1.5-7B over 500 images shows the ratio of Visual Uncertainty to Contextual Prior hallucination is approximately 2:1.

Based on these findings, the authors propose a training-free Detect–Mitigate framework with two main components:

Three stages:

1. Object Token Identification: Using the observation that object tokens exhibit significantly higher image attention than non-object tokens, they compute average image attention A img(ot) within the Image-Attention Stage and threshold it (τ attn = 0.15) to identify object tokens.

2. Semantic Consistency Check: Select Top-k attended image tokens (k = 3), apply Logit Lens to decode each, and compare semantic similarity via WordNet with threshold τ sim = 0.8. If consistent → Real; if inconsistent → Hallucination.

3. Hallucination Classification: Mask high-attention regions and regenerate. If the hallucinated token disappears → Visual Uncertainty; if it persists → Contextual Prior.

High-Attention Regions Masking (HARM) for visual uncertainty hallucination: By removing the high-uncertainty evidence, this strategy effectively reduces hallucination of this type. The masked image is I mask = (1 − M) ⊙ I + M ⊙ µ, where M is a binary mask over the top-k attended patches and µ is a replacement value (e.g., mean color).

Visual Evidence Enhanced Decoding (VEED) for contextual prior hallucination: Since the high-attention regions can still be decoded to the correct tokens via Logit Lens, but are suppressed during generation due to the strong priors, the authors inject the true visual evidence from these regions into the generation process. They extract visual logits via Logit Lens from the most attended region, obtain masking logits from the masked-image model, and fuse: p t = softmax((1 − α)·z t vis + α·z t mask), where α controls the strength of visual-evidence injection.

On the COCO-based hallucination detection task with LLaVA-1.5-7B, LLCC achieves the best results: Precision 0.7870, Recall 0.7955, F1 0.7932, outperforming Uncertainty Score (F1 0.6182), InterConf (F1 0.6811), and SVAR (F1 0.6842). The authors argue baselines fail because they rely on decoding probabilities or cross-layer aggregates, weakening their causal link to the image or focus on attention 'quantity' over 'quality' from non-image-dominant layers. LLCC uniquely locks onto the visual source for object tokens and then checks if that source provides consistent semantic support, and can additionally identify different causal types of hallucinations, which prior methods cannot.

Evaluated on CHAIR and AMBER benchmarks across LLaVA-1.5-7B, LLaVA-1.5-13B, Shikra-7B, and Qwen2-VL-7B, compared against Greedy, Beam, Nucleus sampling, VCD, OPERA, DeCo, Devils, and PAI:

• CHAIR: The method achieves the largest drops across all four models, e.g., on LLaVA-1.5-7B reducing CHAIRS from Greedy's 49.8 to 26.8 and CHAIRI from 20.4 to 10.0; on Shikra-7B reaching 31.4 CHAIRS and 12.7 CHAIRI; on Qwen2-VL-7B achieving 24.0 CHAIRS and 8.3 CHAIRI.

• AMBER: The method obtains the lowest hallucination rates across models while keeping strong coverage. On LLaVA-1.5-7B: CHAIR 2.8, Cover 51.2, Hal 14.7, Cog 1.2.

Notably, most baselines decrease on Cover rate, indicating that they sacrifice valid content to mitigate hallucination, whereas our approach achieves the lowest hallucination rates without eroding the original expression. The authors attribute this to the two-stage framework: "explicit object-token localization and cause-aware hallucination detection, and targeted mitigation: masking only a minimal set of image tokens and applying decoding enhancement only for the second hallucination type, thereby limiting impact on original content."

The paper's stated contributions are:

1. Revealing the same attention, different truth phenomenon, showing the key of object hallucination is the semantic inconsistency between high-attention visual features and the text output, not the attention strength, and categorizing object hallucination into two core mechanisms: visual uncertainty and contextual prior.

2. Proposing a simple yet effective training-free 'Detect-Mitigate' framework comprising the Logit-Lens Consistency Check for detection and two targeted mitigation strategies (HARM and VEED).

3. Achieving SOTA performance on multiple object hallucination benchmarks across several LVLMs, proving its effectiveness and generalizability.

1. Detect object hallucinations online during decoding.

• Identify object tokens by their average image attention in the image-dominant layers (e.g., layers 20–27 for LLaVA-1.5-7B, threshold ≈0.15).

• Apply Logit Lens to the top-3 attended image tokens and compare their decoded semantic content with the generated object token using WordNet similarity (threshold ≈0.8).

• Flag the object as hallucinated when the attended visual evidence does not semantically support the token.

• The improved system can catch hallucinations with ~0.79 F1, outperforming uncertainty/confidence-based detectors, and can do so during generation rather than only after full captioning.

2. Distinguish two different hallucination causes and respond differently.

• Mask the high-attention image regions and regenerate the flagged token.

• If the hallucinated token disappears: classify as visual uncertainty hallucination and apply High-Attention Regions Masking (HARM) by replacing those ambiguous/uncertain patches with a neutral mean color.

• If the hallucinated token persists: classify as contextual prior hallucination and apply Visual Evidence Enhanced Decoding (VEED) by injecting Logit-Lens visual logits from the attended region into the softmax fusion, overriding the prior-driven suppression.

• The improved system can reduce hallucinations from 49.8 CHAIRS to 26.8 on LLaVA-1.5-7B, and from 24.0 to 8.3 CHAIRS on Qwen2-VL-7B, while addressing each failure mode causally instead of applying one generic fix.

3. Mitigate hallucinations without sacrificing valid caption content.

• Use explicit object-token localization so mitigation touches only object-related tokens and a minimal set of high-attention image patches.

• Apply HARM only for visual-uncertainty cases and VEED only for contextual-prior cases; do not globally perturb attention or decoding distributions.

• The improved system maintains or improves coverage and valid-content retention on AMBER and CHAIR, unlike many sampling methods that reduce hallucination by making captions shorter or vaguer.

4. Provide interpretable, evidence-based explanations for each object mention.

• For every object token, output whether the attended visual regions decode to consistent semantics, and which hallucination mechanism is responsible.

• The improved system can tell users: “The word ‘bowl’ is unsupported because the attended region is a blurry ceramic-like patch,” or “The word ‘microwave’ is a contextual-prior error because the attended region actually decodes to a stove.”

• This enables debugging, trust auditing, and targeted correction of LVLM behavior.

5. Plug into existing LVLMs without retraining.

• Implement the Detect–Mitigate framework as a training-free inference wrapper compatible with LLaVA-1.5, LLaVA-1.5-13B, Shikra-7B, and Qwen2-VL-7B, and usable with greedy, beam, or nucleus decoding.

• The improved system can be deployed on top of already-released checkpoints, improving their reliability and factuality without modifying model weights, and without requiring additional data collection.

Large Vision-Language Models (LVLMs) often suffer from object hallucination, generating objects that are absent from the image. Prior work largely attributes this to insufficient visual attention. However, we find that both real and hallucinated objects receive equally strong visual attention in the model's mid-to-late layers, suggesting that the key issue may not be how much the model attends, but what it attends to and why. To this end, we decode the visual features of high-attention regions using Logit Lens, and observe that regions corresponding to real objects can be correctly decoded to the target object tokens, whereas those for hallucinated objects cannot. Building on this, we identify two hallucination mechanisms: (i) visual uncertainty, triggered by semantically similar or confusable regions; masking these regions eliminates the hallucination. (ii) contextual prior, triggered by strong co-occurrence priors; even when the initially attended region is masked, the hallucination persists and attention drifts to other regions. Based on these findings, we propose a simple yet effective training-free Detect-Mitigate framework comprising a Logit-Lens Consistency Check to detect hallucination and targeted remedies: High-Attention Regions Masking (HARM) for visual uncertainty hallucination, and Visual Evidence Enhanced Decoding (VEED) for contextual prior hallucination. Our approach achieves state-of-the-art results on multiple hallucination benchmarks. Code will be available.