Benchmarking and Learning Object-Centric Procedural Understanding in Instructional Videos
cs.CV
Submitted: 2025-12-03
Updated: 2026-10-29
Comments: NeurIPS 2026 ED Track
License: http://creativecommons.org/licenses/by/4.0/
The gist: Procedural activities are fundamentally driven by object state transitions, yet existing instructional video benchmarks remain action-centric and cannot evaluate whether models reason about how…
Terminology
Abstract
Procedural activities are fundamentally driven by object state transitions, yet existing instructional video benchmarks remain action-centric and cannot evaluate whether models reason about how objects evolve toward task completion. In this work, we introduce ProcObject-10K, the first benchmark that jointly evaluates object-centric reasoning and temporal evidence grounding in instructional videos, across both egocentric and exocentric views. It comprises 10,522 open-ended VideoQA pairs grounded in 1,799 videos, spanning 137 tasks across 9 domains and five reasoning types covering preconditions, state evolution, counterfactuals, mistakes, and readiness. Benchmarking 13 leading MLLMs reveals a substantial answering-grounding gap: models produce plausible answers while failing to localize the supporting evidence (mIoU < 45%), exposing their reliance on linguistic priors rather than recognizing and reasoning about the object dynamics. As a step toward closing this gap, we further provide an object-centric supervised fine-tuning baseline with pseudo object-level supervision and spatial-temporal constraints. Models fine-tuned on ProcObject-10K not only improve on the benchmark itself, but also transfer effectively to other grounded VideoQA and embodied planning tasks. The dataset, annotations, and evaluation toolkit have been publicly released on our project page to support future research on object-centric procedural understanding: https://wenliangguo.github.io/ProcObject-10K-Page.
Sources
- Qwen3-VL Technical Report
- ProMQA-Assembly: Multimodal Procedural QA Dataset on Assembly
- ProMQA: Question Answering Dataset for Multimodal Procedural Activity Understanding
- Multimodal Subtask Graph Generation from Instructional Videos
- VEDIT: Latent Prediction Architecture For Procedural Video Representation Learning
- SurveillanceVQA-589K: A Benchmark for Comprehensive Surveillance Video-Language Understanding with Large Models
- A Survey on Hallucination in Large Vision-Language Models
- Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detection
- Learning and Verification of Task Structure in Instructional Videos
- SCHEMA: State CHangEs MAtter for Procedure Planning in Instructional Videos
- EgoExOR: An Ego-Exo-Centric Operating Room Dataset for Surgical Activity Understanding
- Meta Reinforcement Learning with Autonomous Inference of Subtask Dependencies
- InternVL3.5: Advancing Open-Source Multimodal Models in Versatility, Reasoning, and Efficiency
- Qwen3 Technical Report
- BERTScore: Evaluating Text Generation with BERT
Related papers
- Do Egocentric Video-Language Models Capture Both Hand- and Object-Centric Cues?
- Out of Sight, Still in Mind: Token Compression for Omni-LLMs
- From Formulas to Protocols: A Compositional View of Binary Target Segmentation Metrics
- Traffic Sign Recognition in Autonomous Driving: Dataset, Benchmark, and Field Experiment
- Efficient All-in-One Weather Restoration using Spectral Harmonization
- Peak-End-Net: A Peak-End Rule Inspired Framework for Generalizable Video Aesthetic Assessment