Don't Pause! Every prediction matters in a streaming video
summary
The gist
Streaming video models should respond the moment an event unfolds, not after the moment has passed The paper introduces SPOT-Bench to evaluate multi-turn proactive queries and Timeliness-F1 to…
In short
The paper introduces SPOT-Bench to test streaming video models for proactive queries, focusing on responding immediately when events unfold rather than after them. It finds that offline models often spam unprompted predictions, and proposes AsynKV, a training-free adaptation that uses persistent query state to suppress unwanted responses during idle times. This shows that proper memory management can improve streaming performance without retraining.
Key concepts
- SPOT-Bench
- A benchmark with 940 videos testing models on detection tasks like identifying action boundaries or critical points in video. It evaluates whether models can target ongoing or future events in real-time, ensuring they are silent unless an event occurs.
- Timeliness-F1 (T-F1)
- A metric that continuously assesses model responses throughout a video stream. It rewards predictions that are both accurate and perfectly timed relative to the event, penalizing both late and early predictions to measure overall streaming quality.
- AsynKV
- A training-free adaptation for streaming models that manages query state using persistent short-term (16 frames) and long-term (256 frames) memory. It allows the model to summarize during idle periods, extending context without increasing response latency.
- Dead-time compute scaling
- A technique where processing tasks, like episodic summarization, are performed during periods when the model is not actively responding to a query. This keeps the streaming response speed fast while still allowing for background memory management.
Terminology used across episodes
This episode discusses
- Don't Pause! Every prediction matters in a streaming video · Paper Radio
- GPT-4 Technical Report
- Qwen3-VL Technical Report
- Qwen2.5-VL Technical Report
- PerceptionLM: Open-Access Data and Models for Detailed Visual Understanding
- ARC-AGI-2: A New Challenge for Frontier AI Reasoning Systems
- Streaming Video Question-Answering with In-context Video KV-Cache Retrieval
- Video-R1: Reinforcing Video Reasoning in MLLMs
- ViSpeak: Visual Instruction Feedback in Streaming Videos
- Human Action Anticipation: A Survey
- LLaVA-OneVision: Easy Visual Task Transfer
- LLaVA-NeXT-Interleave: Tackling Multi-image, Video, and 3D in Large Multimodal Models
- StreamingBench: Assessing the Gap for MLLMs to Achieve Streaming Video Understanding
- Video-ChatGPT: Towards Detailed Video Understanding via Large Vision and Language Models
- LiveVLM: Efficient Online Video Understanding via Streaming-Oriented KV Cache and Retrieval
- video-SALMONN-o1: Reasoning-enhanced Audio-visual Large Language Model
- StreamBridge: Turning Your Offline Video Large Language Model into a Proactive Streaming Assistant
- Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolution
- InternVL3.5: Advancing Open-Source Multimodal Models in Versatility, Reasoning, and Efficiency
- MMDuet2: Enhancing Proactive Interaction of Video MLLMs with Multi-Turn Reinforcement Learning
- ProactiveVideoQA: A Comprehensive Benchmark Evaluating Proactive Interactions in Video Large Language Models
The paper
Don't Pause! Every prediction matters in a streaming video · Read on arXiv
National University of Singapore · Google Inc.
Transcript
Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.
Tom: I'm Tom.
Jane: And I'm Jane, here with Lu, Meng and our in-house large language model, Lalam.
Tom: We're looking at the paper: "Don't Pause! Every prediction matters in a streaming video".
Jane: Here's the gist of the paper. Streaming video models should respond the moment an event unfolds, not after the moment has passed The paper introduces SPOT-Bench to evaluate multi-turn proactive queries and Timeliness-F1 to measure streaming predictions across the entire video horizon This benchmark reveals that offline models detect events reliably but spam predictions unprompted, and that training-free adaptation like AsynKV can improve streaming behavior SPOT-Bench is a multi-turn, multi-response benchmark comprising 940 videos spanning 68 hours of egocentric and exocentric footage At its core, SPOT-Bench centers on detection tasks that evaluate the fundamental perception skills required for timely, real-time interactions The benchmark evaluates models under realistic streaming conditions with the goals of targeting ongoing and future events rather than past ones, demonstrating timeliness while remaining silent otherwise.
Tom: Evaluation Protocol and Metrics The evaluation protocol requires every prediction to be assessed continuously at every frame, penalizing incorrect or mistimed predictions throughout the video stream To this end, the authors introduce Timeliness-F1 (T-F1), a consolidated streaming evaluation metric that continuously assesses model responses along three axes: timeliness, multi-turn instruction following, and response frequency.
Jane: This metric rewards well-timed predictions across the entire video while penalizing over- and under-responses.
Tom: Model Performance Findings The benchmark reveals several key findings regarding model behavior under streaming inference (i) offline models detect events reliably but spam predictions unprompted, and (ii) post-training for silence reduces spamming but induces unresponsiveness.
Paper summary: Lu: The core idea here is that evaluation needs to change from just scoring a few points in time to assessing continuous performance throughout the whole video stream Lu They set up SPOT-Bench with nine hundred forty videos spanning sixty-eight hours of footage, focusing on detection tasks that test the basic perception skills needed for real-time interaction.
Meng: So it's testing if the model can detect things while it’s happening, not just look at a snapshot later?
Lu: Right. And they use these detection tasks alongside interaction and intervention tasks to simulate real-world scenarios, like helping someone cook a recipe or navigating around obstacles.
Jane: That moves beyond simple question answering into actual assistive capabilities.
Tom: And the crucial part is how they measure it, because they introduce Timeliness-F1, which is this consolidated metric that assesses timeliness, multi-turn instruction following, and response frequency all at once across every single frame.
Meng: So we're not just checking if the answer is right; we're checking *when* it was right and how often the model starts talking when it shouldn't.
Lu: And they define this timing using human-calibrated gold intervals, which gives them a concrete way to know what "on time" means for different kinds of tasks Jane But the finding they highlight is that offline models are good at detecting events reliably, but they tend to spam predictions when nothing relevant happens Tom That’s a big detail because it tells us that even if the model sees something, it might still be noisy and unhelpful if it doesn't know when to stop talking.
Conclusion: Tom: So, looking at the title "Don't Pause! Every prediction matters in a streaming video," what I see is that the authors are pushing for a complete overhaul of how we test these models for real-time use.
Jane: It really boils down to demanding that models operate on the moment an event occurs, which requires us to move away from fixed-point evaluation methods.
Lu: The implication is that if you want a model to be a useful assistant, it can't just be accurate; it has to be temporally aware and disciplined about when it speaks.
Meng: From a deployment standpoint, that means we need metrics like Timeliness-F1 so we can actually gauge if the model is behaving responsibly when running in production.
Tom: And they’re pointing out that training-free adaptation methods, like AsynKV, can help reduce those unwanted spam predictions without needing to retrain the whole model from scratch.
Jane: It opens up a direction for improving streaming models by focusing on managing that persistent query state so the model knows when to be quiet.
More episodes
- 2603.28595-Optimistic Actor-Critic with Parametric Policies for Linear Markov Decision Processes
- 2603.29765-Training-Free Dynamic Upcycling of Expert Language Models
- 2605.00778-Observable Performance Does Not Fully Reflect Adaptive System Organization: A Multi-Level Analysis of Gait Dynamics Under Occlusal Constraint
- 2605.06906-TraXion: Rethinking Pre-training Frameworks for Mobility and Beyond
- 2605.09165-Sparse Layers are Critical to Scaling Looped Language Models
- 2605.09439-Inverse Design for Conditional Distribution Matching
- 2605.10313-Signature Approach for Contextual Bandits with Nonlinear and Path-dependent Rewards: Regret Bound and Statistical Inference
- 2605.10878-Minimum Neural Weight Norm = Kolmogorov Complexity
- 2605.11617-MIST: Reliable Streaming Decision Trees for Online Class-Incremental Learning via McDiarmid Bound
- 2605.15085-From Data to Action: Accelerating Refinery Optimization with AI