JEPA-x: Cross-Predictive Physics Grounding for Forecastable Latent Dynamics
cs.LG
Submitted: 2026-08-25
Updated: 2026-08-28
Comments: Under review
License: http://creativecommons.org/licenses/by/4.0/
The gist: Latent world models plan by predicting how candidate actions advance learned latent dynamics.
Terminology
Abstract
Latent world models plan by predicting how candidate actions advance learned latent dynamics. In self-predictive models, however, the encoder and predictor are optimized jointly and can co-adapt to latent transitions that are easy to predict but weakly constrained by the physical evolution of the scene. We introduce the cross-predictive JEPA (JEPA-x), which grounds latent dynamics in privileged physical trajectories. JEPA-x treats visual observations and physical states as corresponding views of the same action-conditioned trajectory, advances both through a shared predictor, and matches each prediction to the future representations of both modalities. This encourages the action-conditioned predictor to learn a common transition rule across the two views. Privileged physical state is used only during training, leaving a visual-only model at deployment. Empirical results show that JEPA-x reduces the rollout drift of a newly fitted predictor from 0.361 to 0.104 and increases mean control success from 53.6% to 78.2% on a multi-task suite spanning six evaluation subfamilies. We additionally show that direct physical-state regression improves decodability without improving forecastability or control, indicating that the benefit comes from shaping latent dynamics rather than merely encoding physical variables.
Sources
- LeJEPA: Provable and Scalable Self-Supervised Learning Without the Heuristics
- MC-JEPA: A Joint-Embedding Predictive Architecture for Self-Supervised Learning of Motion and Content Features
- Revisiting Feature Prediction for Learning Visual Representations from Video
- PIGDreamer: Privileged Information Guided World Models for Safe Partially Observable Reinforcement Learning
- Pri4R: Learning World Dynamics for Vision-Language-Action Models with Privileged 4D Representation
- LeWorldModel: Stable End-to-End Joint-Embedding Predictive Architecture from Pixels
- Phys-JEPA: Physics-Informed Latent World Models for Multivariate Time-Series Forecasting
- DeepMind Control Suite
- PhyLatent: Learning Dynamics-Relevant Representations for JEPA World Models
- DINO-WM: World Models on Pre-trained Visual Features enable Zero-shot Planning
Related papers
- Polynomial-Augmented Neural Networks (PANNs) with Weak Orthogonality Constraints for Enhanced Function and PDE Approximation
- AIRL-S: Unifying Reinforcement Learning and Search-Based Test-Time Scaling via Adversarial Inverse Reinforcement Learning
- Transformers as Bayesian In-Context Experimenters: Smoothness-Adaptive Efficient ATE Estimation
- Convergence issues in Relational Concept Analysis based on AOC-posets
- Beliefs Beyond Posteriors: The Predictive Cost of ELBO Binding
- Understanding Diffusion Models via Ratio-Based Function Approximation with SignReLU Networks