Don't Pause! Every prediction matters in a streaming video

arXiv:2604.24317 · cs.CV · Submitted 2026-04-27 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: I'm Tom.

Jane: And I'm Jane, here with Lu, Meng and our in-house large language model, Lalam.

Tom: We're looking at the paper: "Don't Pause! Every prediction matters in a streaming video".

Jane: Here's the gist of the paper. Streaming video models should respond the moment an event unfolds, not after the moment has passed The paper introduces SPOT-Bench to evaluate multi-turn proactive queries and Timeliness-F1 to measure streaming predictions across the entire video horizon This benchmark reveals that offline models detect events reliably but spam predictions unprompted, and that training-free adaptation like AsynKV can improve streaming behavior SPOT-Bench is a multi-turn, multi-response benchmark comprising 940 videos spanning 68 hours of egocentric and exocentric footage At its core, SPOT-Bench centers on detection tasks that evaluate the fundamental perception skills required for timely, real-time interactions The benchmark evaluates models under realistic streaming conditions with the goals of targeting ongoing and future events rather than past ones, demonstrating timeliness while remaining silent otherwise.

Tom: Evaluation Protocol and Metrics The evaluation protocol requires every prediction to be assessed continuously at every frame, penalizing incorrect or mistimed predictions throughout the video stream To this end, the authors introduce Timeliness-F1 (T-F1), a consolidated streaming evaluation metric that continuously assesses model responses along three axes: timeliness, multi-turn instruction following, and response frequency.

Jane: This metric rewards well-timed predictions across the entire video while penalizing over- and under-responses.

Tom: Model Performance Findings The benchmark reveals several key findings regarding model behavior under streaming inference (i) offline models detect events reliably but spam predictions unprompted, and (ii) post-training for silence reduces spamming but induces unresponsiveness.

Paper summary: Lu: The core idea here is that evaluation needs to change from just scoring a few points in time to assessing continuous performance throughout the whole video stream Lu They set up SPOT-Bench with nine hundred forty videos spanning sixty-eight hours of footage, focusing on detection tasks that test the basic perception skills needed for real-time interaction.

Meng: So it's testing if the model can detect things while it’s happening, not just look at a snapshot later?

Lu: Right. And they use these detection tasks alongside interaction and intervention tasks to simulate real-world scenarios, like helping someone cook a recipe or navigating around obstacles.

Jane: That moves beyond simple question answering into actual assistive capabilities.

Tom: And the crucial part is how they measure it, because they introduce Timeliness-F1, which is this consolidated metric that assesses timeliness, multi-turn instruction following, and response frequency all at once across every single frame.

Meng: So we're not just checking if the answer is right; we're checking *when* it was right and how often the model starts talking when it shouldn't.

Lu: And they define this timing using human-calibrated gold intervals, which gives them a concrete way to know what "on time" means for different kinds of tasks Jane But the finding they highlight is that offline models are good at detecting events reliably, but they tend to spam predictions when nothing relevant happens Tom That’s a big detail because it tells us that even if the model sees something, it might still be noisy and unhelpful if it doesn't know when to stop talking.

Conclusion: Tom: So, looking at the title "Don't Pause! Every prediction matters in a streaming video," what I see is that the authors are pushing for a complete overhaul of how we test these models for real-time use.

Jane: It really boils down to demanding that models operate on the moment an event occurs, which requires us to move away from fixed-point evaluation methods.

Lu: The implication is that if you want a model to be a useful assistant, it can't just be accurate; it has to be temporally aware and disciplined about when it speaks.

Meng: From a deployment standpoint, that means we need metrics like Timeliness-F1 so we can actually gauge if the model is behaving responsibly when running in production.

Tom: And they’re pointing out that training-free adaptation methods, like AsynKV, can help reduce those unwanted spam predictions without needing to retrain the whole model from scratch.

Jane: It opens up a direction for improving streaming models by focusing on managing that persistent query state so the model knows when to be quiet.

National University of Singapore · Google Inc.

cs.CV

Submitted: 2026-04-27

Updated: 2026-11-05

Comments: NeurIPS 2026; https://dibschat.github.io/SPOT-Bench

License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/

Importance score: 91/100

The gist: Streaming video models should respond the moment an event unfolds, not after the moment has passed The paper introduces SPOT-Bench to evaluate multi-turn proactive queries and Timeliness-F1 to…

Key concepts

SPOT-Bench
A benchmark with 940 videos testing models on detection tasks like identifying action boundaries or critical points in video. It evaluates whether models can target ongoing or future events in real-time, ensuring they are silent unless an event occurs.
Timeliness-F1 (T-F1)
A metric that continuously assesses model responses throughout a video stream. It rewards predictions that are both accurate and perfectly timed relative to the event, penalizing both late and early predictions to measure overall streaming quality.
AsynKV
A training-free adaptation for streaming models that manages query state using persistent short-term (16 frames) and long-term (256 frames) memory. It allows the model to summarize during idle periods, extending context without increasing response latency.
Dead-time compute scaling
A technique where processing tasks, like episodic summarization, are performed during periods when the model is not actively responding to a query. This keeps the streaming response speed fast while still allowing for background memory management.

Terminology

Summary

Streaming video models should respond the moment an event unfolds, not after the moment has passed The paper introduces SPOT-Bench to evaluate multi-turn proactive queries and Timeliness-F1 to measure streaming predictions across the entire video horizon This benchmark reveals that offline models detect events reliably but spam predictions unprompted, and that training-free adaptation like AsynKV can improve streaming behavior

Introduction to the Benchmark

SPOT-Bench is a multi-turn, multi-response benchmark comprising 940 videos spanning 68 hours of egocentric and exocentric footage At its core, SPOT-Bench centers on detection tasks that evaluate the fundamental perception skills required for timely, real-time interactions The benchmark evaluates models under realistic streaming conditions with the goals of targeting ongoing and future events rather than past ones, demonstrating timeliness while remaining silent otherwise.

Evaluation Protocol and Metrics

The evaluation protocol requires every prediction to be assessed continuously at every frame, penalizing incorrect or mistimed predictions throughout the video stream To this end, the authors introduce Timeliness-F1 (T-F1), a consolidated streaming evaluation metric that continuously assesses model responses along three axes: timeliness, multi-turn instruction following, and response frequency. This metric rewards well-timed predictions across the entire video while penalizing over- and under-responses. p. 2

Model Performance Findings

The benchmark reveals several key findings regarding model behavior under streaming inference (i) offline models detect events reliably but spam predictions unprompted, and (ii) post-training for silence reduces spamming but induces unresponsiveness. The authors propose AsynKV, a training-free streaming adaptation that tracks query-response state explicitly across frames to suppress repetitive responses without retraining. p. 2

Proposed Solution and Future Directions

AsynKV is designed to resolve the issue where offline MLLMs respond unprompted or when nothing relevant occurs, by introducing Persistent Query State and Long-short term KV memory This adaptation allows AsynKV to run summarization and memory management asynchronously during dead-time, extending temporal context without affecting response latency. The work points toward the need for higher-quality streaming instruction tuning data for assistive and agentic scenarios and opens an unexplored axis for improving streaming models through dead-time compute scaling. p. 0

Task Taxonomy and Annotation

The benchmark includes six streaming tasks grouped into Detection, Interaction, and Intervention categories These include Action Boundary Detection (ABD) to identify temporal boundaries, Point-of-No-Return Detection (PNR) for critical temporal points, Streaming Question Answering (SQA), Streaming Procedure Guidance (SPG), Solicited Intervention (SI), and Unsolicited Intervention (UI). Gold intervals derived from a controlled human study define the temporal range of annotator agreement, which is adopted as the ground truth for response timeliness. p. 7 The data collection pipeline involves semi-automated annotation where LLMs assist human annotators in generating queries and assigning timestamps based on source dataset metadata. p. 5

Human Calibration of Timeliness

To capture temporal ambiguity, a controlled human study was conducted to establish gold intervals for response timeliness across tasks. The resulting intervals show that agreement correlates with task granularity, with Detection tasks showing the highest agreement due to their precise temporal definitions. p. 7 The Timeliness-score (T-score) is defined using a weighting function centered on the human-calibrated gold interval [ts, te] to penalize both early and late predictions asymmetrically. p. 7

Evaluation Metric Details

Timeliness-F1 captures exactly what is required by evaluating every prediction across the full video horizon, rewarding timely, accurate predictions while penalizing both over- and under-responses. The greedy matching procedure uses an occupancy budget K to control how many temporally valid, semantically correct predictions each slot may collect before closing. p. 7 Precision and recall are computed using the sum of these weighted TPs, and their harmonic mean defines the final Timeliness-F1. p. 8

AsynKV Implementation

AsynKV maintains a short-term KV of NS=16 frames on GPU and long-term KV storing up to NL=256 frames on CPU for memory management It retrieves the top-K=4 relevant frames from long-term memory when a query is posed, preserving the arrow of time. Dead-time compute scaling is exploited by performing episodic summarization during idle periods, which runs off the critical path and keeps streaming response latency bounded. p. 2 This approach demonstrates that with proper KV management, streaming inference need not compromise the perception quality. p. 2

Conclusion on Adaptation

The results suggest that with proper KV management, streaming inference need not compromise the perception quality AsynKV achieves state-of-the-art on both proactive and retrospective VideoQA benchmarks, reinforcing the claim that training-free adaptation can suffice for streaming inference. The paper concludes that SPOT-Bench provides a foundation for future research toward general-purpose, always-on, real-time assistants. p. 0 The clear gap between closed- and open-source models on Interaction and Intervention tasks highlights the need for higher quality streaming instruction tuning data. p. 2 The discovery of deadtime in streaming inference opens a largely unexplored axis for improving streaming models without affecting response latency. p. 2

Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.

Improvements for AI systems

  1. Develop models that can operate continuously under partial observations by implementing AsynKV, which tracks query-response state explicitly across frames, suppressing repetitive responses without retraining. This allows for long-short term KV memory to be utilized efficiently during dead-time compute, extending temporal context without affecting response latency.

  2. Introduce a new evaluation protocol called Timeliness-F1 to assess streaming predictions by their temporal precision and balanced coverage across the entire video, rewarding well-timed predictions while penalizing both over- and under-responses. This metric captures behaviors previously unquantified, such as response echoing and posttraining-induced under-response.

  3. Design systems capable of proactive intervention by focusing on safety scenarios. The benchmark tasks like Solicited Intervention (SI) and Unsolicited Intervention (UI) force models to recognize imminent risk and intervene precisely at the critical moment, early enough to prevent failure, yet restrained enough to avoid unnecessary or disruptive responses.

  4. Enhance general event perception in streaming settings by training models with a silence objective during post-training. This modification is shown to reduce over-responses (↓ false positives) while maintaining event perception quality, even though it may induce unresponsiveness (↑ false negatives).

Sources

Related papers