Stress Tests REVEAL Fragile Temporal and Visual Grounding in Video-Language Models
cs.CV
Submitted: 2026-02-11
Updated: 2026-09-16
License: http://creativecommons.org/licenses/by/4.0/
The gist: Video-Language Models (VidLMs) achieve strong benchmark scores, yet these scores often hide whether models use the video at all.
Terminology
Abstract
Video-Language Models (VidLMs) achieve strong benchmark scores, yet these scores often hide whether models use the video at all. We show that VidLM failures follow two pathways: some visual signals are never reliably encoded, while others are encoded but overridden by model priors. We introduce REVEAL, a diagnostic stress-test benchmark for quantifying when and why VidLMs under-use visual evidence. REVEAL contains five controlled probes: camera-motion sensitivity, cross-frame integration, video sycophancy, language-only shortcuts, and temporal expectation bias. Together, they test whether models encode basic video signals, combine evidence across frames, and preserve visual evidence against user assertions, language cues, and learned event expectations. Across 12 VidLMs we find systematic failures along both pathways, with most models falling below chance on the binary and six-way probes that humans solve at 78--100% accuracy. Under assertive prompts, a model's output distribution becomes nearly invariant to whether it is shown a real video or random noise, making visual evidence effectively causally inert. We further carry out mechanistic probes to identify where these failures arise in the model pipeline and why visual evidence is lost. REVEAL provides a scalable, human-verified framework for moving beyond aggregate scores toward structured, reproducible evaluation of multimodal reliability.
Sources
- Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilities
- Lost in Time: A New Temporal Benchmark for VideoLLMs
- SycEval: Evaluating LLM Sycophancy
- FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning
- Mitigating Hallucination in Large Vision-Language Models via Adaptive Attention Calibration
- Qwen2.5-VL Technical Report
- TemporalBench: Benchmarking Fine-grained Temporal Understanding for Multimodal Video Models
- VidHal: Benchmarking Temporal Hallucinations in Vision LLMs
- A Survey on LLM-as-a-Judge
- Benchmarking Neural Network Robustness to Common Corruptions and Perturbations
- Measuring Sycophancy of Language Models in Multi-turn Dialogues
- Have the VLMs Lost Confidence? A Study of Sycophancy in VLMs
- MotionBench: Benchmarking and Improving Fine-grained Video Motion Understanding for Vision Language Models
- VideoHallu: Evaluating and Mitigating Multi-modal Hallucinations on Synthetic Video Understanding
- Towards Understanding Camera Motions in Any Video
- TRUTH DECAY: Quantifying Multi-Turn Sycophancy in Language Models
- VideoComp: Advancing Fine-Grained Compositional and Temporal Alignment in Video-Text Models
- Exploring Ordinal Bias in Action Recognition for Instructional Videos
- RTime-QA: A Benchmark for Atomic Temporal Event Understanding in Large Multi-modal Models
- VLind-Bench: Measuring Language Priors in Large Vision-Language Models
Related papers
- Loss Knows Best: Detecting Annotation Errors in Videos via Loss Trajectories
- AnchorWeave: World-Consistent Video Generation with Retrieved Local Spatial Memories
- Benchmarking the Robustness of Foundation Models for Mammography under Domain Shift
- MambaX-Net: Dual-Input Mamba-Enhanced Cross-Attention Network for Longitudinal MRI Segmentation
- TeleOCR: Navigating Document Parsing Across Digital and Camera-Captured Documents
- A Survey on Efficient Vision-Language-Action Models