Don't Pause! Every prediction matters in a streaming video
Listen
Radio episode about this paper
Transcript
Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.
Tom: I'm Tom.
Jane: And I'm Jane, here with Lu, Meng and our in-house large language model, Lalam.
Tom: We're looking at the paper: "Don't Pause! Every prediction matters in a streaming video".
Jane: Here's the gist of the paper. Streaming video models should respond the moment an event unfolds, not after the moment has passed The paper introduces SPOT-Bench to evaluate multi-turn proactive queries and Timeliness-F1 to measure streaming predictions across the entire video horizon This benchmark reveals that offline models detect events reliably but spam predictions unprompted, and that training-free adaptation like AsynKV can improve streaming behavior SPOT-Bench is a multi-turn, multi-response benchmark comprising 940 videos spanning 68 hours of egocentric and exocentric footage At its core, SPOT-Bench centers on detection tasks that evaluate the fundamental perception skills required for timely, real-time interactions The benchmark evaluates models under realistic streaming conditions with the goals of targeting ongoing and future events rather than past ones, demonstrating timeliness while remaining silent otherwise.
Tom: Evaluation Protocol and Metrics The evaluation protocol requires every prediction to be assessed continuously at every frame, penalizing incorrect or mistimed predictions throughout the video stream To this end, the authors introduce Timeliness-F1 (T-F1), a consolidated streaming evaluation metric that continuously assesses model responses along three axes: timeliness, multi-turn instruction following, and response frequency.
Jane: This metric rewards well-timed predictions across the entire video while penalizing over- and under-responses.
Tom: Model Performance Findings The benchmark reveals several key findings regarding model behavior under streaming inference (i) offline models detect events reliably but spam predictions unprompted, and (ii) post-training for silence reduces spamming but induces unresponsiveness.
Paper summary: Lu: The core idea here is that evaluation needs to change from just scoring a few points in time to assessing continuous performance throughout the whole video stream Lu They set up SPOT-Bench with nine hundred forty videos spanning sixty-eight hours of footage, focusing on detection tasks that test the basic perception skills needed for real-time interaction.
Meng: So it's testing if the model can detect things while it’s happening, not just look at a snapshot later?
Lu: Right. And they use these detection tasks alongside interaction and intervention tasks to simulate real-world scenarios, like helping someone cook a recipe or navigating around obstacles.
Jane: That moves beyond simple question answering into actual assistive capabilities.
Tom: And the crucial part is how they measure it, because they introduce Timeliness-F1, which is this consolidated metric that assesses timeliness, multi-turn instruction following, and response frequency all at once across every single frame.
Meng: So we're not just checking if the answer is right; we're checking *when* it was right and how often the model starts talking when it shouldn't.
Lu: And they define this timing using human-calibrated gold intervals, which gives them a concrete way to know what "on time" means for different kinds of tasks Jane But the finding they highlight is that offline models are good at detecting events reliably, but they tend to spam predictions when nothing relevant happens Tom That’s a big detail because it tells us that even if the model sees something, it might still be noisy and unhelpful if it doesn't know when to stop talking.
Conclusion: Tom: So, looking at the title "Don't Pause! Every prediction matters in a streaming video," what I see is that the authors are pushing for a complete overhaul of how we test these models for real-time use.
Jane: It really boils down to demanding that models operate on the moment an event occurs, which requires us to move away from fixed-point evaluation methods.
Lu: The implication is that if you want a model to be a useful assistant, it can't just be accurate; it has to be temporally aware and disciplined about when it speaks.
Meng: From a deployment standpoint, that means we need metrics like Timeliness-F1 so we can actually gauge if the model is behaving responsibly when running in production.
Tom: And they’re pointing out that training-free adaptation methods, like AsynKV, can help reduce those unwanted spam predictions without needing to retrain the whole model from scratch.
Jane: It opens up a direction for improving streaming models by focusing on managing that persistent query state so the model knows when to be quiet.
National University of Singapore · Google Inc.
cs.CV
Submitted: 2026-04-27
Updated: 2026-11-05
Comments: NeurIPS 2026; https://dibschat.github.io/SPOT-Bench
License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
Importance score: 91/100
The gist: Streaming video models should respond the moment an event unfolds, not after the moment has passed The paper introduces SPOT-Bench to evaluate multi-turn proactive queries and Timeliness-F1 to…
Key concepts
- SPOT-Bench
- A benchmark with 940 videos testing models on detection tasks like identifying action boundaries or critical points in video. It evaluates whether models can target ongoing or future events in real-time, ensuring they are silent unless an event occurs.
- Timeliness-F1 (T-F1)
- A metric that continuously assesses model responses throughout a video stream. It rewards predictions that are both accurate and perfectly timed relative to the event, penalizing both late and early predictions to measure overall streaming quality.
- AsynKV
- A training-free adaptation for streaming models that manages query state using persistent short-term (16 frames) and long-term (256 frames) memory. It allows the model to summarize during idle periods, extending context without increasing response latency.
- Dead-time compute scaling
- A technique where processing tasks, like episodic summarization, are performed during periods when the model is not actively responding to a query. This keeps the streaming response speed fast while still allowing for background memory management.
Terminology
Summary
Streaming video models should respond the moment an event unfolds, not after the moment has passed The paper introduces SPOT-Bench to evaluate multi-turn proactive queries and Timeliness-F1 to measure streaming predictions across the entire video horizon This benchmark reveals that offline models detect events reliably but spam predictions unprompted, and that training-free adaptation like AsynKV can improve streaming behavior
Introduction to the Benchmark
SPOT-Bench is a multi-turn, multi-response benchmark comprising 940 videos spanning 68 hours of egocentric and exocentric footage At its core, SPOT-Bench centers on detection tasks that evaluate the fundamental perception skills required for timely, real-time interactions The benchmark evaluates models under realistic streaming conditions with the goals of targeting ongoing and future events rather than past ones, demonstrating timeliness while remaining silent otherwise.
Evaluation Protocol and Metrics
The evaluation protocol requires every prediction to be assessed continuously at every frame, penalizing incorrect or mistimed predictions throughout the video stream To this end, the authors introduce Timeliness-F1 (T-F1), a consolidated streaming evaluation metric that continuously assesses model responses along three axes: timeliness, multi-turn instruction following, and response frequency. This metric rewards well-timed predictions across the entire video while penalizing over- and under-responses. p. 2
Model Performance Findings
The benchmark reveals several key findings regarding model behavior under streaming inference (i) offline models detect events reliably but spam predictions unprompted, and (ii) post-training for silence reduces spamming but induces unresponsiveness. The authors propose AsynKV, a training-free streaming adaptation that tracks query-response state explicitly across frames to suppress repetitive responses without retraining. p. 2
Proposed Solution and Future Directions
AsynKV is designed to resolve the issue where offline MLLMs respond unprompted or when nothing relevant occurs, by introducing Persistent Query State and Long-short term KV memory This adaptation allows AsynKV to run summarization and memory management asynchronously during dead-time, extending temporal context without affecting response latency. The work points toward the need for higher-quality streaming instruction tuning data for assistive and agentic scenarios and opens an unexplored axis for improving streaming models through dead-time compute scaling. p. 0
Task Taxonomy and Annotation
The benchmark includes six streaming tasks grouped into Detection, Interaction, and Intervention categories These include Action Boundary Detection (ABD) to identify temporal boundaries, Point-of-No-Return Detection (PNR) for critical temporal points, Streaming Question Answering (SQA), Streaming Procedure Guidance (SPG), Solicited Intervention (SI), and Unsolicited Intervention (UI). Gold intervals derived from a controlled human study define the temporal range of annotator agreement, which is adopted as the ground truth for response timeliness. p. 7 The data collection pipeline involves semi-automated annotation where LLMs assist human annotators in generating queries and assigning timestamps based on source dataset metadata. p. 5
Human Calibration of Timeliness
To capture temporal ambiguity, a controlled human study was conducted to establish gold intervals for response timeliness across tasks. The resulting intervals show that agreement correlates with task granularity, with Detection tasks showing the highest agreement due to their precise temporal definitions. p. 7 The Timeliness-score (T-score) is defined using a weighting function centered on the human-calibrated gold interval [ts, te] to penalize both early and late predictions asymmetrically. p. 7
Evaluation Metric Details
Timeliness-F1 captures exactly what is required by evaluating every prediction across the full video horizon, rewarding timely, accurate predictions while penalizing both over- and under-responses. The greedy matching procedure uses an occupancy budget K to control how many temporally valid, semantically correct predictions each slot may collect before closing. p. 7 Precision and recall are computed using the sum of these weighted TPs, and their harmonic mean defines the final Timeliness-F1. p. 8
AsynKV Implementation
AsynKV maintains a short-term KV of NS=16 frames on GPU and long-term KV storing up to NL=256 frames on CPU for memory management It retrieves the top-K=4 relevant frames from long-term memory when a query is posed, preserving the arrow of time. Dead-time compute scaling is exploited by performing episodic summarization during idle periods, which runs off the critical path and keeps streaming response latency bounded. p. 2 This approach demonstrates that with proper KV management, streaming inference need not compromise the perception quality. p. 2
Conclusion on Adaptation
The results suggest that with proper KV management, streaming inference need not compromise the perception quality AsynKV achieves state-of-the-art on both proactive and retrospective VideoQA benchmarks, reinforcing the claim that training-free adaptation can suffice for streaming inference. The paper concludes that SPOT-Bench provides a foundation for future research toward general-purpose, always-on, real-time assistants. p. 0 The clear gap between closed- and open-source models on Interaction and Intervention tasks highlights the need for higher quality streaming instruction tuning data. p. 2 The discovery of deadtime in streaming inference opens a largely unexplored axis for improving streaming models without affecting response latency. p. 2
Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.
Improvements for AI systems
-
Develop models that can operate continuously under partial observations by implementing AsynKV, which
tracks query-response state explicitly across frames, suppressing repetitive responses without retraining.
This allows forlong-short term KV
memory to be utilized efficiently during dead-time compute, extending temporal context without affecting response latency. -
Introduce a new evaluation protocol called Timeliness-F1 to assess streaming predictions by their
temporal precision and balanced coverage across the entire video,
rewarding well-timed predictions while penalizing bothover- and under-responses.
This metric captures behaviors previously unquantified, such asresponse echoing
andposttraining-induced under-response.
-
Design systems capable of proactive intervention by focusing on safety scenarios. The benchmark tasks like Solicited Intervention (SI) and Unsolicited Intervention (UI) force models to
recognize imminent risk and intervene precisely at the critical moment, early enough to prevent failure, yet restrained enough to avoid unnecessary or disruptive responses.
-
Enhance general event perception in streaming settings by training models with a
silence objective
during post-training. This modification is shown to reduce over-responses (↓ false positives
) while maintaining event perception quality, even though it mayinduce unresponsiveness (↑ false negatives).
Sources
- GPT-4 Technical Report
- Qwen3-VL Technical Report
- Qwen2.5-VL Technical Report
- PerceptionLM: Open-Access Data and Models for Detailed Visual Understanding
- ARC-AGI-2: A New Challenge for Frontier AI Reasoning Systems
- Streaming Video Question-Answering with In-context Video KV-Cache Retrieval
- Video-R1: Reinforcing Video Reasoning in MLLMs
- ViSpeak: Visual Instruction Feedback in Streaming Videos
- Human Action Anticipation: A Survey
- LLaVA-OneVision: Easy Visual Task Transfer
- LLaVA-NeXT-Interleave: Tackling Multi-image, Video, and 3D in Large Multimodal Models
- StreamingBench: Assessing the Gap for MLLMs to Achieve Streaming Video Understanding
- Video-ChatGPT: Towards Detailed Video Understanding via Large Vision and Language Models
- LiveVLM: Efficient Online Video Understanding via Streaming-Oriented KV Cache and Retrieval
- video-SALMONN-o1: Reasoning-enhanced Audio-visual Large Language Model
- StreamBridge: Turning Your Offline Video Large Language Model into a Proactive Streaming Assistant
- Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolution
- InternVL3.5: Advancing Open-Source Multimodal Models in Versatility, Reasoning, and Efficiency
- MMDuet2: Enhancing Proactive Interaction of Video MLLMs with Multi-Turn Reinforcement Learning
- ProactiveVideoQA: A Comprehensive Benchmark Evaluating Proactive Interactions in Video Large Language Models
Related papers
- TEA: Text Encoder Alignment for Robust Concept Erasure in Text-to-Image Models
- Do Egocentric Video-Language Models Capture Both Hand- and Object-Centric Cues?
- Aura: Consistent Multi-Subject Video Generation via VLM-Grounded Semantic Alignment
- PVRF: All-in-one Adverse Weather Removal via Prior-modulated and Velocity-constrained Rectified Flow
- DLWM: Diverse Latent World Models for Efficient Multimodal Reasoning
- Out of Sight, Still in Mind: Token Compression for Omni-LLMs