LRC-JEPA: Disentangling Dynamics and Residual Context for Efficient World Models
cs.LG, cs.AI, cs.RO
Submitted: 2026-09-28
Updated: 2026-09-28
License: http://creativecommons.org/licenses/by-nc-nd/4.0/
The gist: Compact JEPA world models enable efficient latent-space planning, but low-dimensional representation trained under reward-free self-supervision must encode both action-conditioned dynamics and
Terminology
Abstract
Compact JEPA world models enable efficient latent-space planning, but low-dimensional representation trained under reward-free self-supervision must encode both action-conditioned dynamics and predictable visual context. This competition can entangle controllable state with high-rank nuisance appearance and degrade planning as scenes become more complex. We introduce LRC-JEPA, a lightweight end-to-end world model that routes information into a compact predictive latent z and learned-query residual-context embeddings u. Only z is propagated by the dynamics model and used for planning, while u captures temporally persistent information for cross-attention reconstruction; a differentiable residual connection encourages the latent to retain complementary dynamic content. Under explicit assumptions, we show that the resulting representation is sufficient, minimal, nuisance-invariant, and disentangled. Across four simulated control environments, LRC-JEPA improves average planning success over a parameter-matched JEPA baseline by 9 percentage points and matches or exceeds substantially larger pretrained models. On the real-world Bridge-v2 set, its 5.5M-parameter active encoder outperforms DINO-WM (22.1M) and V-JEPA2 (303.9M) encoders while also enabling faster planning. Physical-state probes, reconstruction interventions, and ablations confirm the effectiveness of LRC-JEPA's representation disentanglement.
Sources
- Deep Variational Information Bottleneck
- V-JEPA 2: Self-Supervised Video Models Enable Understanding, Prediction and Planning
- LeJEPA: Provable and Scalable Self-Supervised Learning Without the Heuristics
- MC-JEPA: A Joint-Embedding Predictive Architecture for Self-Supervised Learning of Motion and Content Features
- SAM 3: Segment Anything with Concepts
- Isolating Sources of Disentanglement in Variational Autoencoders
- A Generalization Theory for JEPA-Based World Models
- An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
- Visual Foresight: Model-Based Deep Reinforcement Learning for Vision-Based Robotic Control
- World Models
- When Does LeJEPA Learn a World Model?
- Temporally Centered SIGReg Improves LeWorldModel Representations for Robot Policy Learning
- LeWorldModel: Stable End-to-End Joint-Embedding Predictive Architecture from Pixels
- Causal-JEPA: Learning World Models through Object-Level Latent Masking
- Cosmos World Foundation Model Platform for Physical AI
- Scalable Diffusion Models with Transformers
- Joint Embedding Predictive Architectures Focus on Slow Features
- A Lightweight Library for Energy-Based Joint-Embedding Predictive Architectures
- Subspace-Decomposed JEPAs: Disentangling Progression and Content in Latent World Models
- On the Identifiability of Controlled World Models
Related papers
- Polynomial-Augmented Neural Networks (PANNs) with Weak Orthogonality Constraints for Enhanced Function and PDE Approximation
- AIRL-S: Unifying Reinforcement Learning and Search-Based Test-Time Scaling via Adversarial Inverse Reinforcement Learning
- Transformers as Bayesian In-Context Experimenters: Smoothness-Adaptive Efficient ATE Estimation
- Convergence issues in Relational Concept Analysis based on AOC-posets
- Beliefs Beyond Posteriors: The Predictive Cost of ELBO Binding
- Understanding Diffusion Models via Ratio-Based Function Approximation with SignReLU Networks