Language-Augmented Video Action Anticipation: Design Fundamentals, Benchmarks, and Open Challenges
cs.CV
Submitted: 2026-09-15
Updated: 2026-09-30
Comments: 29 pages, 5 figures, 19 tables. Review article. Supplementary material, machine-readable data, and public artifacts are available at https://github.com/mahsa7290/language-augmented-action-anticipation
License: http://creativecommons.org/licenses/by-nc-nd/4.0/
The gist: Action anticipation predicts future human actions from partial video under incomplete context and temporal uncertainty.
Terminology
Abstract
Action anticipation predicts future human actions from partial video under incomplete context and temporal uncertainty. Recent systems introduce large language models (LLMs), vision-language models (VLMs), or language-derived semantics at different stages, but reported gains are difficult to interpret when task formulation, visual pretraining, supervision, decoder design, and evaluation code change simultaneously. The central contribution of this review is an evidence-aware design map that crosses task regime with the point at which language-derived information intervenes. We characterise task regimes along six axes. These axes organise the literature into five broad task families: single-action, sequence, object-interaction, cross-view, and planning-oriented settings. C1-C3 locate interventions in context construction, goal/intention modelling, and future decoding, while C4 is treated as an adjacent, emerging grounding/executability extension. Unlike a generic processing pipeline, the map links each intervention to an appropriate counterfactual, failure diagnosis, and permissible evidence claim. Supporting contributions include a protocol-level audit of Ego4D-LTA and EPIC-KITCHENS-100, a multidimensional evidence profile, and the Backbone-Aware Comparison and Ablation Protocol (BCAP). The unresolved EK-100 record is treated as a reporting-comparability case study and is not used as a leaderboard. Evidence for LLM benefits, goal ambiguity, and horizon effects is therefore formulated as testable hypotheses requiring matched validation, not as causal conclusions. The accompanying package contains the coded evidence, source locators, protocol metadata, and versioned catalogue used in the review.
Sources
- A Short Note on the Kinetics-700-2020 Human Action Dataset
- UCF101: A Dataset of 101 Human Actions Classes From Videos in The Wild
- A Survey on Deep Learning Techniques for Action Anticipation
- Human Action Anticipation: A Survey
- V-JEPA 2: Self-Supervised Video Models Enable Understanding, Prediction and Planning
- V-JEPA 2.1: Unlocking Dense Features in Video Self-Supervised Learning
- On the Efficacy of Text-Based Input Modalities for Action Anticipation
- Vision and Intention Boost Large Language Model in Long-Term Action Anticipation
- VideoLLM: Modeling Video Sequence with Large Language Models
- LEAP: LLM-Generation of Egocentric Action Programs
- TR-LLM: Integrating Trajectory Data for Scene-Aware LLM-Based Human Action Prediction
- Anticipate & Act : Integrating LLMs and Classical Planning for Efficient Task Execution in Household Environments
- Hierarchical Planning with Latent World Models
- StreamMind: Unlocking Full Frame Rate Streaming Video Dialogue through Event-Gated Cognition
- Memory-efficient Streaming VideoLLMs for Real-time Procedural Video Understanding
- Streaming Video Question-Answering with In-context Video KV-Cache Retrieval
- SparseVLM: Visual Token Sparsification for Efficient Vision-Language Model Inference
- Video Token Sparsification for Efficient Multimodal LLMs in Autonomous Driving
- PruneVid: Visual Token Pruning for Efficient Video Large Language Models
- AdaCM$^2$: On Understanding Extremely Long-Term Video with Adaptive Cross-Modality Memory Reduction
Related papers
- Loss Knows Best: Detecting Annotation Errors in Videos via Loss Trajectories
- AnchorWeave: World-Consistent Video Generation with Retrieved Local Spatial Memories
- Benchmarking the Robustness of Foundation Models for Mammography under Domain Shift
- MambaX-Net: Dual-Input Mamba-Enhanced Cross-Attention Network for Longitudinal MRI Segmentation
- TeleOCR: Navigating Document Parsing Across Digital and Camera-Captured Documents
- A Survey on Efficient Vision-Language-Action Models