Streaming Interventions: Can Video Large Language Models Correct Mistakes as They Occur?
cs.CV, cs.LG
Submitted: 2026-06-08
Updated: 2026-11-11
Comments: To appear at NeurIPS 2026 (E&D) and the project page is available at https://apratimbh.github.io/livecookv2/
License: http://creativecommons.org/licenses/by-nc-nd/4.0/
The gist: Learning everyday skills, like cooking a dish, relies increasingly on instructional media such as online videos.
Terminology
Abstract
Learning everyday skills, like cooking a dish, relies increasingly on instructional media such as online videos. This opens the door to the use of video (and multimodal) large language models (LLMs) as task guidance assistants. A crucial capability for the real-world success of a prospective task guidance assistant is it's ability to intervene proactively as soon as a mistake is apparent in order to guide the user. To evaluate this crucial capability, we introduce Ego-MC-Bench (Mistake Corrections), a benchmark for evaluating reactive, step-by-step task guidance in realistic cooking scenarios. Extensive experiments show that Ego-MC-Bench is highly challenging for state-of-the-art video LLMs. We argue that a key reason is the limited availability of training data for fine-tuning models on this task. Although there exists a wide range of cooking video datasets, existing datasets lack examples of mistakes along with appropriately timed interventions. To help address this data limitation, we also introduce Ego-CoMist, a counterfactual synthetic dataset created by transforming non-interactive cooking videos into supervised training examples showing proactive interventions. We show that fine-tuning on Ego-CoMist yields performance gains especially for smaller and more efficient video LLMs that are well suited for delivering assistance on edge devices.
Sources
- GPT-4 Technical Report
- StreamMind: Unlocking Full Frame Rate Streaming Video Dialogue through Event-Gated Cognition
- Qwen3 Technical Report
- Qwen3-VL Technical Report
- Qwen2.5-VL Technical Report
- The Llama 3 Herd of Models
- Ego-Exo4D: Understanding Skilled Human Activity from First- and Third-Person Perspectives
- Video-MMMU: Evaluating Knowledge Acquisition from Multi-Discipline Professional Videos
- OVO-Bench: How Far is Your Video-LLMs from Real-World Online Video Understanding?
- StreamingBench: Assessing the Gap for MLLMs to Achieve Streaming Video Understanding
- LiveVLM: Efficient Online Video Understanding via Streaming-Oriented KV Cache and Retrieval
- A Simple Baseline for Streaming Video Understanding
- Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilities
- Gemini: A Family of Highly Capable Multimodal Models
- Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context
- Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolution
- InternVL3.5: Advancing Open-Source Multimodal Models in Versatility, Reasoning, and Efficiency
- Towards Top-Down Reasoning: An Explainable Multi-Agent Approach for Visual Question Answering
- Streaming Video Instruction Tuning
- StreamingVLM: Real-Time Understanding for Infinite Video Streams
Related papers
- TEA: Text Encoder Alignment for Robust Concept Erasure in Text-to-Image Models
- Do Egocentric Video-Language Models Capture Both Hand- and Object-Centric Cues?
- Aura: Consistent Multi-Subject Video Generation via VLM-Grounded Semantic Alignment
- PVRF: All-in-one Adverse Weather Removal via Prior-modulated and Velocity-constrained Rectified Flow
- DLWM: Diverse Latent World Models for Efficient Multimodal Reasoning
- Out of Sight, Still in Mind: Token Compression for Omni-LLMs