What Drives Success in Physical Planning with Joint-Embedding Predictive World Models?
cs.AI, cs.LG, cs.RO, stat.ML
Submitted: 2025-12-30
Updated: 2026-09-02
Comments: V2 of the article: - Added AdaLN-zero - Added table comparing JEPA-WMs with baselines with std translating per-seed variability only, no variability across epochs - Reordered figures in main body of the paper V3: added data scaling experiments, theoretical appendix section on autoregressive rollout, acceptance at TMLR V4: Added funding acknowledgements for Jean Ponce
Code: https://github.com/facebookresearch/jepa-wms
License: http://creativecommons.org/licenses/by-nc-nd/4.0/
The gist: A long-standing challenge in AI is to develop agents capable of solving a wide range of physical tasks and generalizing to new, unseen tasks and environments.
Terminology
Abstract
A long-standing challenge in AI is to develop agents capable of solving a wide range of physical tasks and generalizing to new, unseen tasks and environments. A popular recent approach involves training a world model from state-action trajectories and subsequently use it with a planning algorithm to solve new tasks. Planning is commonly performed in the input space, but a recent family of methods has introduced planning algorithms that optimize in the learned representation space of the world model, with the promise that abstracting irrelevant details yields more efficient planning. In this work, we characterize models from this family as JEPA-WMs and investigate the technical choices that make algorithms from this class work. We propose a comprehensive study of several key components with the objective of finding the optimal approach within the family. We conducted experiments using both simulated environments and real-world robotic data, and studied how the model architecture, the training objective, and the planning algorithm affect planning success. We combine our findings to propose a model that outperforms two established baselines, DINO-WM and V-JEPA-2-AC, in both navigation and manipulation tasks. Code, data and checkpoints are available at https://github.com/facebookresearch/jepa-wms.
Sources
- Cosmos World Foundation Model Platform for Physical AI
- Combating the Compounding-Error Problem with a Multi-step Model
- Back to the Features: DINO as a Foundation for Video World Models
- VaViM and VaVAM: Autonomous Driving through Video Generative Modeling
- Scheduled Sampling for Sequence Prediction with Recurrent Neural Networks
- $\pi_0$: A Vision-Language-Action Flow Model for General Robot Control
- D4RL: Datasets for Deep Data-Driven Reinforcement Learning
- Embodied AI Agents: Modeling the World
- The CMA Evolution Strategy: A Tutorial
- GAIA-1: A Generative World Model for Autonomous Driving
- $\pi^{*}_{0.6}$: a VLA That Learns From Experience
- $\pi_{0.5}$: a Vision-Language-Action Model with Open-World Generalization
- Closing the Train-Test Gap in World Models for Gradient-Based Planning
- On the difficulty of training Recurrent Neural Networks
- A Reduction of Imitation Learning and Structured Prediction to No-Regret Online Learning
- Proximal Policy Optimization Algorithms
- DINOv3
- A Lightweight Library for Energy-Based Joint-Embedding Predictive Architectures
- JEPA-Bisim: Learning Robust Visual Representations for Planning with Joint-Embedding Predictive World Models
- DDP-WM: Disentangled Dynamics Prediction for Efficient World Models
Related papers
- MAVEN-T: Reinforced Heterogeneous Distillation for Real-Time Multi-Agent Trajectory Prediction
- Model Discovery Agent: LLM-assisted Bayesian experiment design for data-efficient discovery of mechanistic world models
- Agents' Last Exam
- The Clinician's Veto: Navigating Trust, Liability, and Uncertainty in Autonomous AI Prescribing
- MindHelper: Closed-Loop Embodied Mental-State Reasoning for Precision Intervention
- Incumbent Advantage: Brand Bias and Cognitive Manipulation Dynamics in LLM Recommendation Systems