SplitJEPA: Learning Invariant and Variant Latent Worlds without Reconstruction

The gist: SplitJEPA introduces a reconstruction-free Joint Embedding Predictive Architecture (JEPA) that jointly recovers the latent state and its invariant and variant organization directly in representation space without any reconstruction, proving that this structure can be identified under stationary Gaussian predictive dynamics and a full-rank variation condition

Problem Background

Learning a useful representation of a dynamical world requires more than preserving information about its observations; the representation should also reveal how the latent state is structured into factors that remain stable and factors that vary across relevant changes This objective has inspired invariant representation learning from multiview and temporal robotic data (Sermanet et al., 2018; Nair et al., 2022), as well as content-style approaches that seek to separate shared information from varying factors Approaches relying on reconstruction introduce additional modeling and computational costs and can direct representation capacity toward noise and observation details that are unnecessary for downstream tasks

SplitJEPA Mechanism

SplitJEPA models the latent state directly through prediction in representation space without reconstruction, yet no existing result recovers the invariant and variant parts of the state they learn The key insight is that prediction in representation space and consistency across invariant pairs resolve distinct levels of latent ambiguity Specifically, it achieves prediction in latent space, while block consistency under sufficient variation resolves the mixing between invariant and variant factors The method establishes conditions under which SplitJEPA identifies both latent blocks up to within-block orthogonal transformations, without introducing an observation decoder

Identification Process

The identification process is achieved in two stages:

1. Predictive Identification and Remaining Ambiguity: JEPA learns from statistical relations between observations without reconstructing them, and under the predictive setting, its identifiability analysis characterizes global latent recovery through a linear representation This leads to the conclusion that predictive identification alone does not determine the invariant-variant partition because a global rotation may still mix the two latent blocks

2. Identifying the Partition through Variation: Once globally linear, agreement across correspondences constrains its dependence on the variant state, and this constraint is formalized by Assumption 1 (Sufficient Variation), which ensures that the secondmoment matrix of the variant changes satisfies ΣΔL = EDinv ΔzLΔz⊤ L ≻ 0 This positive definiteness forces the cross-block mixing matrices, such as A12 and A21, to be zero under sufficient variation

Theoretical Guarantees

The paper proves that the learned representation decomposes into invariant and variant subspaces up to independent within-block orthogonal transformations without reconstruction Theorem 3 further guarantees operational robustness, stating that for a downstream predictor depending solely on the invariant representation rH, the prediction is locally insensitive to perturbations in zL because its Jacobian satisfies ∂rH/dzL = 0

Experimental Validation

Experiments on synthetic nonlinear systems and robotic manipulation tasks support the theoretical results and show their practical value for both robustness and efficiency In synthetic evaluations, SplitJEPA achieves near-perfect block recovery with little cross-block leakage compared to other methods like LeJEPA Robotic experiments demonstrate that the invariant representation rH retains much of the information needed for task performance, and the variant block rL responds more strongly to execution-specific factors Efficiency results show that SplitJEPA reaches the recovery target in all five runs with a 2.70× lower capped runtime compared to V3

Conclusion

SplitJEPA resolves two levels of ambiguity, and the recovered structure isolates computations based on the invariant block from variant perturbations The results suggest that representation learning can recover not only the information in a latent world but also how it is organized The framework demonstrates that the invariant and variant subspaces are identified up to independent within-block isometries, without requiring an observation decoder or access to the ground-truth latent variables

--- Page 1 ---

SplitJEPA: Learning Invariant and Variant Latent Worlds without Reconstruction Ruijin Hua1, Zichuan Liu2, Zhuokai Zhao3, Yujia Zheng1 1 University of Illinois Urbana-Champaign 2 Carnegie Mellon University 3 University of Chicago Abstract Understanding a dynamical world calls for more than a latent state that summarizes its observations: the state should also be organized into the factors that stay shared across related observations and the factors that vary between them For example, a robot pushing a cube to a goal should take the same action when the camera shifts or the lights dim, since nothing in the scene has moved Existing approaches to this decomposition commonly obtain it through reconstruction, so the latent variables must first explain the entire observational world before their organization can be trusted Joint embedding predictive architectures (JEPAs) model the latent state directly and never reconstruct, yet no existing result recovers the invariant and variant parts of the state they learn How to learn the invariant-variant structure of the latent world without paying for its reconstruction therefore remains open To close this gap, we introduce SplitJEPA, a JEPA that jointly recovers the latent state and its invariant and variant organization directly in representation space, without any reconstruction We prove that, under stationary Gaussian predictive dynamics and a full-rank variation condition, SplitJEPA identifies the invariant and variant subspaces up to independent block-wise isometries, without introducing an observation decoder Since the guarantee needs no decoder, the result extends reconstruction-free latent recovery to invariant-variant block identification Experiments on synthetic nonlinear systems and robotic manipulation tasks support the theoretical results and show their practical value for both robustness and efficiency

--- Page 2 ---

Introduction Learning a useful representation of a dynamical world requires more than preserving information about its observations The representation should also reveal how the latent state is structured into factors that remain stable and factors that vary across relevant changes (Scholkopf et al., 2021; ¨K¨ugelgen et al., 2021) Robot manipulation provides an example Different executions of the same task may vary in robot configuration, local motion, viewpoint, illumination, and object appearance A useful representation should retain the invariant content while separating the sources of variation, allowing downstream computations to use stable information without discarding variation when it is useful This motivates structured representation learning: recovering not only the global latent state, but also its invariant–variant decomposition This objective has inspired invariant representation learning from multiview and temporal robotic data (Sermanet et al., 2018; Nair et al., 2022), as well as content-style approaches that seek to separate shared information from varying factors (Shrestha & Fu, 2025; Timilsina et al., 2026; Wang et al., 2025) These approaches commonly rely on reconstruction or related generative objectives: an encoder maps observations into latent variables, while the decoder requires those variables to keep enough information to reproduce the observations Although effective, reconstruction introduces additional modeling and computational costs and can direct representation capacity toward noise and observation details that are unnecessary for downstream tasks More fundamentally, reconstruction encourages the representation to preserve information needed to reproduce the observations, but it does not by itself specify how that information should be partitioned into invariant and variant structure Indeed, an invertible nonlinear reparameterization may preserve the complete observation distribution while mixing the underlying latent factors (Hyvarinen & Pajunen, 1999; Locatello et al., ¨2019) Identifiable generative models address this ambiguity by introducing additional supervision

--- Page 3 ---

Reconstruction-Based Ours: Reconstruction-Free Encoder Observations Encoder Identifiable Blocks Observation or functional constraints Decoder Robust Figure 1: Block identifiability without observation reconstruction. Reconstruction-based identifiable models recover structured latent factors through an encoder-decoder pathway together with additional structural assumptions SplitJEPA combines prediction in latent space with block consistency to recover invariant and variant structure directly in representation space or functional constraints (Hyvarinen et al., 2024; Moran & Aragam, 2026) Our goal is to explore whether suitable structure can identify the desired decomposition directly in representation space without requiring an observation decoder Reconstruction-free learning offers a promising foundation Joint-Embedding Predictive Architectures (JEPAs) learn informative representations by making predictions directly in representation space without reconstruction (Assran et al., 2023; Bardes et al., 2024; Dawid & LeCun, 2024) Related robotic studies further show that useful invariances can be learned without pixel reconstruction and that separating view-invariant from view-dependent information can improve visual world models (Zhang et al., 2021; Pang et al.

1. The SplitJEPA architecture can be used to generate invariant representations for robotic manipulation tasks by training it on matched trajectories, allowing downstream policies to operate robustly against viewpoint and lighting changes without affecting task execution performance (the invariant block remains comparatively stable when viewpoint and execution factors change).

2. A system utilizing the learned representation-only interface, specifically Ours-rH, can be designed for high-reliability control where predictions are locally insensitive to perturbations in zL because the representation is decomposed into blocks such that the prediction is locally insensitive to perturbations in zL: ∂Y/∂zL = 0.

3. A downstream predictor relying solely on the invariant representation can be deployed for tasks where the shared state information is paramount, as gH(z) = A11zH implies that all admissible states with the same zH produce the same invariant representation, even when their observations differ substantially.