Imagine Before You Predict: Interleaved Latent Visual Reasoning for Video Event Prediction
cs.CV
Submitted: 2026-06-04
Updated: 2026-10-15
Comments: Accepted to EMNLP 2026 Main Conference. Code: https://github.com/OpenGVLab/Future-L1
Code: https://github.com/OpenGVLab/Future-L1
License: http://creativecommons.org/licenses/by/4.0/
The gist: Video event prediction (VEP) requires models to infer unobserved future states from partial video evidence.
Terminology
Abstract
Video event prediction (VEP) requires models to infer unobserved future states from partial video evidence. Existing video MLLMs usually verbalize intermediate future reasoning in text space: once visual evidence is verbalized, fine-grained motion, geometry, and interaction cues can be lost, leading to plausible but visually ungrounded hallucinations. We introduce Future-L1, an interleaved latent visual reasoning framework that lets an MLLM alternate between language tokens and continuous latent visual spans during autoregressive decoding. To train this capability, we construct Future-L1-50K by selecting examples where future visual hints help prediction and align latent states to future-frame embeddings, then further optimize sampled latent trajectories with LA-DAPO, a latent-aware RL objective with outcome-contrastive and temporal-diversity rewards. Future-L1 achieves new state-of-the-art results on both benchmarks: on FutureBench, it improves Qwen3-VL-8B from 61.0 to 85.4 and exceeds the previous best Video-CoE by 10.4 points; on TwiFF-Bench, it improves the average score from 2.44 to 3.04. These results suggest that future-oriented video reasoning benefits from preserving intermediate visual semantics in latent space rather than translating every reasoning step into text.
Sources
- LLaVA-OneVision-2: Towards Next-Generation Perceptual Intelligence
- Qwen3-VL Technical Report
- Janus-Pro: Unified Multimodal Understanding and Generation with Data and Model Scaling
- TEMPURA: Temporal Event Masked Prediction and Understanding for Reasoning in Action
- Video-as-Answer: Predict and Generate Next Video Event with Joint-GRPO
- HyLaR: Hybrid Latent Reasoning with Decoupled Policy Optimization
- Video-MME: The First-Ever Comprehensive Evaluation Benchmark of Multi-modal LLMs in Video Analysis
- Video-MME-v2: Towards the Next Stage in Benchmarks for Comprehensive Video Understanding
- Training Large Language Models to Reason in a Continuous Latent Space
- GLM-5V-Turbo: Toward a Native Foundation Model for Multimodal Agents
- VKnowU: Evaluating Visual Knowledge Understanding in Multimodal LLMs
- Latent Visual Reasoning
- Imagine while Reasoning in Space: Multimodal Visualization-of-Thought
- Learning Goal-Oriented Vision-and-Language Navigation with Self-Improving Demonstrations at Scale
- VideoChat-R1: Enhancing Spatio-Temporal Perception via Reinforcement Fine-Tuning
- TwiFF (Think With Future Frames): A Large-Scale Dataset for Dynamic Visual Reasoning
- VideoAuto-R1: Video Auto Reasoning via Thinking Once, Answering Twice
- Xiaomi OneVL: One-Step Latent Reasoning and Planning with Vision-Language Explanation
- Multimodal Chain of Continuous Thought for Latent-Space Reasoning in Vision-Language Models
- Chain-of-Visual-Thought: Teaching VLMs to See and Think Better with Continuous Visual Tokens
Related papers
- Loss Knows Best: Detecting Annotation Errors in Videos via Loss Trajectories
- AnchorWeave: World-Consistent Video Generation with Retrieved Local Spatial Memories
- Benchmarking the Robustness of Foundation Models for Mammography under Domain Shift
- MambaX-Net: Dual-Input Mamba-Enhanced Cross-Attention Network for Longitudinal MRI Segmentation
- TeleOCR: Navigating Document Parsing Across Digital and Camera-Captured Documents
- A Survey on Efficient Vision-Language-Action Models