STORM-Bench: Evaluating Online Video QA under Evolving and Incomplete Evidence
cs.CV
Submitted: 2026-09-25
Updated: 2026-09-25
Comments: 50 pages, 19 figures, 27 tables
Code: https://github.com/siruzhong/STORM-Bench
License: http://creativecommons.org/licenses/by/4.0/
The gist: Reliable online video question answering requires tracking state transitions while selectively abstaining when visual evidence is insufficient.
Terminology
Abstract
Reliable online video question answering requires tracking state transitions while selectively abstaining when visual evidence is insufficient. Existing benchmarks focus on static recognition or long-range retrieval, rarely evaluating these coupled capabilities under evolving and incomplete evidence. We present STORM-Bench, comprising 5,736 questions across 630 compact, change-dense episodes spanning five egocentric domains (STORM-Real) and two controlled simulation subsets (STORM-Sim) at 1 FPS. Questions are stratified by a proxy for accumulated change intensity (Low, Medium, High) and query-time answerability (Known, Uncertain). To measure reliability, we introduce STORM-BR, a harmonic metric over joint answer-status correctness that exposes abstention failures masked by aggregate accuracy, alongside STORM-BR-ATTR for uncertainty attribution. Across 14 video LLMs, online accuracy peaks at 60.3% (mean 51.7%), whereas STORM-BR ranges from 5.7% to 35.6% (mean 18.8%), driven by pervasive overconfidence on uncertain queries. STORM-Bench shows that task accuracy masks these gaps in epistemic reliability and state tracking. Benchmark and code are available at https://github.com/siruzhong/STORM-Bench.
Sources
- Qwen2.5-VL Technical Report
- Qwen3-VL Technical Report
- Eagle 2.5: Boosting Long-Context Post-Training for Frontier Vision-Language Models
- EGOSTREAM: A Diagnostic Benchmark for Streaming Episodic Memory in Egocentric Vision
- GLM-4.5V and GLM-4.1V-Thinking: Towards Versatile Multimodal Reasoning with Scalable Reinforcement Learning
- AI2-THOR: An Interactive 3D Environment for Visual AI
- OVO-S-Bench: A Hierarchical Benchmark for Streaming Spatial Intelligence in Multimodal LLMs
- StreamingBench: Assessing the Gap for MLLMs to Achieve Streaming Video Understanding
- InternVL3.5: Advancing Open-Source Multimodal Models in Versatility, Reasoning, and Efficiency
- InternVideo2.5: Empowering Video MLLMs with Long and Rich Context Modeling
- MiniCPM-V 4.5: Cooking Efficient MLLMs via Architecture, Data, and Training Recipe
- VideoLLaMA 3: Frontier Multimodal Foundation Models for Image and Video Understanding
- Flash-VStream: Memory-Based Real-Time Understanding for Long Video Streams
Related papers
- Loss Knows Best: Detecting Annotation Errors in Videos via Loss Trajectories
- AnchorWeave: World-Consistent Video Generation with Retrieved Local Spatial Memories
- Benchmarking the Robustness of Foundation Models for Mammography under Domain Shift
- MambaX-Net: Dual-Input Mamba-Enhanced Cross-Attention Network for Longitudinal MRI Segmentation
- TeleOCR: Navigating Document Parsing Across Digital and Camera-Captured Documents
- A Survey on Efficient Vision-Language-Action Models