WorldFlow: Joint Video and 3D Motion Prediction via Large-Scale Pretraining
cs.CV, cs.AI, cs.RO
Submitted: 2026-10-29
Updated: 2026-11-17
Project page: https://worldflow-rbs.github.io
License: http://creativecommons.org/licenses/by/4.0/
The gist: Predicting future physical states stands as a central challenge in embodied intelligence.
Abstract
Predicting future physical states stands as a central challenge in embodied intelligence. Existing video world models leave 3D dynamics implicit, action-based formulations remain tied to specific robot configurations, and 3D motion prediction models use flow-only supervision, providing a limited learning signal for physical dynamics. To circumvent these limitations, we introduce, a generative world model that jointly predicts future video and query-based 3D trajectories. By parameterizing 3D motion as continuous displacement tokens aligned with the video grid, a unified Mixture-of-Transformers (MoT) backbone jointly denoises both modalities under a rectified-flow objective. This formulation couples explicit geometric dynamics with visual evolution, effectively unlocking the physical commonsense embedded within pretrained video models. To empirically scale and validate this unified architecture, we train on a heterogeneous mixture of 1.44 million object-level samples spanning simulated and real-world domains. We further introduce WorldFlowBench, comprising 1,678 evaluation samples from 11 data sources across simulated and real-world interactions, including spatial reasoning tasks. Across all three evaluation sets, WorldFlow achieves the lowest average displacement error (ADE) and final displacement error (FDE) among the evaluated learning-based methods. Notably, on the challenging SpatialCog set, WorldFlow achieves an ADE of 0.060 m, a 54.2% reduction compared with the strongest external baseline. It also reduces ADE and FDE by 55.9% and 68.5%, respectively, compared with its flow-only variant. Project page: https://worldflow-rbs.github.io/.
Sources
- Video-As-Prompt: Unified Semantic Control for Video Generation
- AgiBot World Colosseo: A Large-scale Manipulation Platform for Scalable and Intelligent Embodied Systems
- H3-World: Turning Language Understanding into World Control
- SANA-Video: Efficient Video Generation with Block Linear Diffusion Transformer
- Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilities
- Scaling Egocentric Vision: The EPIC-KITCHENS Dataset
- Emerging Properties in Unified Multimodal Pretraining
- Dream2Flow: Bridging Video Generation and Open-World Manipulation with 3D Object Flow
- T(R,O) Grasp: Efficient Graph Diffusion of Robot-Object Spatial Transformation for Cross-Embodiment Dexterous Grasping
- PointWorld: Scaling 3D World Models for In-The-Wild Robotic Manipulation
- Self Forcing: Bridging the Train-Test Gap in Autoregressive Video Diffusion
- MolmoSpaces: A Large-Scale Open Ecosystem for Robot Navigation and Manipulation
- mu 0: A Scalable 3D Interaction-Trace World Model
- EgoWAM: World Action Models Beyond Pixels with In-the-Wild Egocentric Human Data
- NovaFlow: Zero-Shot Manipulation via Actionable Flow from Generated Videos
- Causal World Modeling for Robot Control
- RoboFlow4D: A Lightweight Flow World Model Toward Real-Time Flow-Guided Robotic Manipulation
- RoboCasa: Large-Scale Simulation of Everyday Tasks for Generalist Robots
- Cosmos 3: Omnimodal World Models for Physical AI
Related papers
- TEA: Text Encoder Alignment for Robust Concept Erasure in Text-to-Image Models
- Do Egocentric Video-Language Models Capture Both Hand- and Object-Centric Cues?
- Aura: Consistent Multi-Subject Video Generation via VLM-Grounded Semantic Alignment
- PVRF: All-in-one Adverse Weather Removal via Prior-modulated and Velocity-constrained Rectified Flow
- DLWM: Diverse Latent World Models for Efficient Multimodal Reasoning
- Out of Sight, Still in Mind: Token Compression for Omni-LLMs