From Suppression to Repair: Mitigating Object Hallucination in Large Vision-Language Models via Localized Distribution Alignment

The gist: ResOT, a training-free method that repairs representations at inference time through localized distribution alignment, substantially reduces object hallucination while improving image caption quality and multimodal performance across multiple benchmarks

Motivation

Object hallucination remains a major obstacle for large vision-language models (LVLMs) to generate reliable content While representation-based methods offer a training-free alternative by modifying the model’s hidden representations, suppressing hallucination-related components can weaken the model’s multimodal capabilities because these components may also contain useful information Therefore, the paper seeks to repair the hallucination-related components rather than discard them.

ResOT Method Overview

ResOT is a training-free method that implements localized repair at inference time through localized distribution alignment During offline calibration, ResOT processes each image with its paired faithful and hallucinated descriptions to collect two sets of hidden representations. It estimates the dominant subspaces of these representations, then projects the dominant hallucinated directions away from the faithful subspace. The leading residual directions define the repair subspace

Key Components of ResOT

The process involves several specific mathematical steps for localization and repair:

ResOT constructs a repair subspace from the dominant hallucinated directions that lie outside the dominant faithful subspace by projecting the hallucinated basis to select intervention directions.

** The residual basis U is obtained, where its columns are orthonormal and satisfy B⊤F U = 0. This ensures that all subsequent updates lie in span(U), leaving components in the dominant faithful subspace unchanged. The paper notes that faithful representations are not confined to their dominant subspace; they still have components in the residual subspace. This motivates localized distribution alignment. The method uses Gaussian optimal transport (OT) to align the hallucinated distribution with the faithful one within this subspace. The resulting map defines repair targets with minimal changes to the original representations. During inference, ResOT adaptively controls how far each token state moves toward its OT target using a likelihood-based gate. The resulting correction is added back to the hidden state along the residual directions, leaving the orthogonal component unchanged. The LVLM weights remain fixed throughout calibration and inference. The theoretical analysis supports the repair subspace and transport target. Gaussian OT provides a closed-form map from the hallucinated distribution to the faithful one, which aligns the fitted residual distributions with minimum expected squared displacement. The optimal map is given by T⋆(z) = ˆµF + A(z − µˆH), where A is determined by the means and covariances of the distributions. During inference, a likelihood-based gate compares the projected state’s likelihood under the two fitted distributions and controls how far it moves toward the target. The resulting correction is added back to the hidden state along the residual directions, leaving the orthogonal component unchanged. The total additional computation per generated token is therefore O(L(dk + k 2)). The paper also notes that the dominant term is O(Ldk) for the total additional computation per generated token. The method retains 81.24% of the original model’s generation throughput while requiring only an additional 8.15 MiB of peak GPU memory. The calibration set analysis shows that ResOT already outperforms Nullu in Cs, BLEU-1, and MME with only 50 calibration pairs. It is also robust to imperfect calibration data as the corrupted samples are split evenly between image–text mismatches and faithful/hallucinated label swaps. The method retains its advantage over Nullu on all three metrics with few calibration pairs and is robust to contamination. The representation distribution visualization shows that the residual subspace exposes a clearer difference between faithful and hallucinated representations than the original and Nullu-style subspaces. This demonstrates that ResOT remains effective with few calibration pairs and is robust to imperfect calibration data. The Mahalanobis KS distances show that the behavior observed at layer 24 is not specific to a single layer or model. The overall results show that ResOT preserves or improves general multimodal performance while reducing hallucination. The total additional computation per generated token is therefore O(Ldk + k 2). The stored residual bases, transport maps, and Gaussian statistics require O(L(dk + k 2)) memory. The method retains 81.24% of the throughput of its paired vanilla run and increases peak GPU memory by 8.15 MiB. The training-free nature means the LVLM weights remain fixed throughout calibration and inference. The authors also note that the method retains its advantage over Nullu on all three metrics with few calibration pairs and is robust to contamination. The residual subspace consistently yields a higher Mahalanobis distance, LDA accuracy, and LDA AUC across all evaluated layers. This demonstrates that ResOT better isolates hallucination-related deviations. The training-free nature means the LVLM weights remain fixed throughout calibration and inference. The method retains its advantage over Nullu on all three metrics with few calibration pairs and is robust to contamination. The training-free nature means the LVLM weights remain fixed throughout calibration and inference. The total additional computation per generated token is therefore O(Ldk + k 2). The stored residual bases, transport maps, and Gaussian statistics require O(L(dk + k 2)) memory. The method retains 81.24% of the throughput of its paired vanilla run and increases peak GPU memory by 8.15 MiB. The training-free nature means the LVLM weights remain fixed throughout calibration and inference. The total additional computation per generated token is therefore O(Ldk + k 2). The stored residual bases, transport maps, and Gaussian statistics require O(L(dk + k 2)) memory. The method retains 81.24% of the throughput of its paired vanilla run and increases peak GPU memory by 8.15 MiB. The training-free nature means the LVLM weights remain fixed throughout calibration and inference. The total additional computation per generated token is therefore O(Ldk + k 2). The stored residual bases, transport maps, and Gaussian statistics require O(L(dk + k 2)) memory. The method retains 81.24% of the throughput of its paired vanilla run and increases peak GPU memory by 8.15 MiB. The training-free nature means the LVLM weights remain fixed throughout calibration and inference.

1. Bold header: Representation-based repair instead of suppression

The system can now repair hallucination-related components by using their distribution in faithful representations to set repair targets, which contrasts with suppression methods that reduce the variation of hallucinated representations along the selected direction, but may also inadvertently alter faithful representations.

2. Bold header: Localized distribution alignment via Gaussian Optimal Transport

The system employs a method where it uses Gaussian optimal transport (OT) to align the hallucinated distribution with the faithful one, defining repair targets by minimizing expected squared displacement in a low-dimensional residual subspace.

3. Bold header: Adaptive inference-time correction

During inference, ResOT adaptively controls how far each token state moves toward its OT target using a likelihood-based gate, which allows the model to selectively apply the correction to avoid uniform shifts across all states.

4. Bold header: Subspace construction based on dominant directions

The method constructs a repair subspace by projecting dominant hallucinated directions away from the faithful subspace, ensuring that the intervention is localized while leaving components in the dominant faithful subspace unchanged.

5. Bold header: Robustness to calibration size and corruption

The system demonstrates robustness, as it performs well with only 50 calibration pairs and remains effective robust to imperfect calibration data even when a fraction of samples are corrupted by image–text mismatches.

6. Bold header: Preservation of multimodal performance

The improved system preserves or even improves image caption quality and broader multimodal performance, achieving gains across benchmarks like MMMU, TextVQA, and ScienceQA while reducing object hallucination.

7. Bold header: Inference efficiency maintenance

ResOT maintains high throughput, retaining 81.24% of the original model’s generation throughput while requiring only an additional 8.15 MiB of peak GPU memory compared to vanilla models under the same hardware configuration.