RECAP: Relation Evidence Calibration for Detecting Spatial Relation Hallucinations in Vision-Language Models
cs.CV
Submitted: 2026-08-04
Updated: 2026-08-04
Comments: 21 pages, 4 figures, including supplementary material. Code: https://github.com/SouthWinter/RECAP
Code: https://github.com/SouthWinter/RECAP
License: http://creativecommons.org/licenses/by/4.0/
The gist: Vision-language models can answer spatial relation questions confidently even when the image supports an incompatible relation.
Terminology
Abstract
Vision-language models can answer spatial relation questions confidently even when the image supports an incompatible relation. We formulate relation-grounded selective prediction: accept or reject an already-produced yes/no answer by auditing its visual support, rather than treating uncertainty as evidence. RECAP, our relation-evidence calibration framework, compares image-conditioned likelihoods for a claim, its semantic contradictions, and optional one-sided supports, then converts these witnesses into an answer-conditioned rejection risk. A calibration-only gate preserves confidence as a veto when confidence is demonstrably informative and otherwise deploys relation evidence alone. Across 20 group/image-disjoint splits, RECAP lowers H-FPR@80 over confidence by between 2.0 and 17.9 points on VSR and raises Acc@80 by 3.0, 8.6, and 12.6 points on What'sUp for Qwen3-VL-8B, InternVL3.5-8B, and LLaVA-1.5-7B. It outperforms matched VCD-style visual contrast on all four primary metrics in all six settings. Full-pool VSR fallback, target-ranked GSR-Bench transfer, equal-budget supervised controls, and two additional checkpoints show a consistent operating principle: structured counterevidence complements certainty when confidence is misaligned, while the gate retains confidence when it is already useful.
Sources
- Qwen3-VL Technical Report
- Predicting When to Trust Vision-Language Models for Spatial Reasoning
- LLaVA-OneVision: Easy Visual Task Transfer
- Mitigating Hallucination in Large Multi-Modal Models via Robust Instruction Tuning
- MMRel: Benchmarking Relation Understanding in Multi-Modal Large Language Models
- GSR-BENCH: A Benchmark for Grounded Spatial Reasoning Evaluation via Multimodal LLMs
- AMBER: An LLM-free Multi-dimensional Benchmark for MLLMs Hallucination Evaluation
- Look Again Before You Abstain:Budgeted Conformal Evidence Acquisition for Reliable Vision-Language Model
- Woodpecker: Hallucination Correction for Multimodal Large Language Models
- VL-Uncertainty: Detecting Hallucination in Large Vision-Language Model via Uncertainty Estimation
Related papers
- Loss Knows Best: Detecting Annotation Errors in Videos via Loss Trajectories
- AnchorWeave: World-Consistent Video Generation with Retrieved Local Spatial Memories
- Benchmarking the Robustness of Foundation Models for Mammography under Domain Shift
- MambaX-Net: Dual-Input Mamba-Enhanced Cross-Attention Network for Longitudinal MRI Segmentation
- TeleOCR: Navigating Document Parsing Across Digital and Camera-Captured Documents
- A Survey on Efficient Vision-Language-Action Models