Predict, Then Retrieve: Cross-Instance Future-State Retrieval from Video Prefixes
Quynh Vo, Thong Nguyen, Vinh-Hien Do, Cong-Duy Nguyen, Anh-Tuan Luu
cs.CV, cs.CL
Submitted: 2026-08-05
Comments: Work in progress
License: http://creativecommons.org/licenses/by-sa/4.0/
The gist: We introduce Predictive State Retrieval (PSR), a task in which a model observes a short video prefix and a temporal question about an object's future state, then retrieves instances from other videos
Terminology
Abstract
We introduce Predictive State Retrieval (PSR), a task in which a model observes a short video prefix and a temporal question about an object's future state, then retrieves instances from other videos or images that depict that state. Unlike action anticipation, which predicts a label, moment retrieval, which localizes an observed event within a video, or video generation, which synthesizes pixels, PSR combines anticipation with cross-instance retrieval across multiple temporal horizons. We construct a benchmark from four datasets with graded, human-validated ground truth, difficulty tiers, and an oracle ceiling. We also propose LFTR, a lightweight retriever with frozen encoders that predicts a question- and horizon-conditioned future latent and matches it in complementary semantic and visual spaces. A ceiling decomposition reveals a clear bottleneck: the true future state is highly retrievable once specified, whereas every predictor we evaluate, including a large multimodal language model with access to the prefix frames, remains far below the oracle. Thus, forecasting rather than perception is the central learnable challenge. LFTR narrows this gap at substantially lower inference cost, and ablations attribute its gains to cross-space fusion and hard-negative training rather than latent rollout. We release the benchmark, code, and evaluation scripts.
Sources
- V-JEPA 2: Self-Supervised Video Models Enable Understanding, Prediction and Planning
- Vision Pretraining for Dense Spatial Perception
- CAST: Modeling Visual State Transitions for Consistent Video Retrieval
- Anticipating Object State Changes in Long Procedural Videos
- STATUS Bench: A Rigorous Benchmark for Evaluating Object State Understanding in Vision-Language Models
- Representation Learning with Contrastive Predictive Coding
- EmbeddingGemma: Powerful and Lightweight Text Representations
- InternVideo: General Video Foundation Models via Generative and Discriminative Learning
Related papers
- Loss Knows Best: Detecting Annotation Errors in Videos via Loss Trajectories
- AnchorWeave: World-Consistent Video Generation with Retrieved Local Spatial Memories
- Benchmarking the Robustness of Foundation Models for Mammography under Domain Shift
- MambaX-Net: Dual-Input Mamba-Enhanced Cross-Attention Network for Longitudinal MRI Segmentation
- TeleOCR: Navigating Document Parsing Across Digital and Camera-Captured Documents
- A Survey on Efficient Vision-Language-Action Models