VidParse: Online Parsing of Egocentric Procedures Like a Pro
cs.CV
Submitted: 2026-08-27
Updated: 2026-10-27
Comments: Accepted at ECCV 2026. v2: GTEA results (Table 1, supplementary Table 1) are now averaged over the four standard splits; v1 reported split 1 only. Implementation details updated to match the released code
Project page: https://learn2phoenix.github.io/VidParse
License: http://creativecommons.org/licenses/by/4.0/
The gist: Translating continuous, noisy egocentric video streams into discrete, temporally ordered action steps is fraught with visual challenges.
Terminology
Abstract
Translating continuous, noisy egocentric video streams into discrete, temporally ordered action steps is fraught with visual challenges. Heavy ego-motion, transient occlusions, and the high intra-class variability of unscripted human-object interactions cause standard frame-level online temporal models to struggle, often resulting in severe over-segmentation and structural collapse. To bridge the gap between unstable low-level perception and high-level procedural logic, we present VidParse, an online, training-free framework that treats activity understanding as a graph-constrained inference problem. Rather than relying on learned temporal filters, we dynamically identify semantic transitions using a temporal similarity matrix over manipulation-anchored features, which are extracted from frozen foundation models to prioritize foreground hand-object interactions. A beam search decoder then leverages an induced procedural task graph to explicitly enforce valid action transitions and prune impossible trajectories. By anchoring robust visual segments to hard procedural constraints, our approach preserves long-range state transitions and achieves up to a 10x improvement in complex multi-step parsing accuracy over strong online baselines, all without requiring a single gradient update.
Sources
- An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
- Training-Free Zero-Shot Temporal Action Detection with Vision-Language Models
- LASER: A Neuro-Symbolic Framework for Learning Spatial-Temporal Scene Graphs with Weak Supervision
- VideoGraph: Recognizing Minutes-Long Human Activities in Videos
- The Kinetics Human Action Video Dataset
- DINOv2: Learning Robust Visual Features without Supervision
- ASFormer: Transformer for Action Segmentation
Related papers
- TEA: Text Encoder Alignment for Robust Concept Erasure in Text-to-Image Models
- Do Egocentric Video-Language Models Capture Both Hand- and Object-Centric Cues?
- Aura: Consistent Multi-Subject Video Generation via VLM-Grounded Semantic Alignment
- PVRF: All-in-one Adverse Weather Removal via Prior-modulated and Velocity-constrained Rectified Flow
- DLWM: Diverse Latent World Models for Efficient Multimodal Reasoning
- Out of Sight, Still in Mind: Token Compression for Omni-LLMs