Spoofing the Latent State: Imperceptible Goal-Image Attacks Derail Planning in LeWorldModel
cs.CV, cs.CR, cs.LG
Submitted: 2026-10-30
Updated: 2026-10-30
Comments: 14 pages, 5 figures, 13 tables
License: http://creativecommons.org/licenses/by-sa/4.0/
The gist: Joint-embedding predictive architectures (JEPAs) have recently emerged as general-purpose visual world models for control.
Terminology
Abstract
Joint-embedding predictive architectures (JEPAs) have recently emerged as general-purpose visual world models for control. A prominent example is LeWorldModel (LeWM), which trains end-to-end from pixels using an isotropic Gaussian anti-collapse regularizer (SIGReg) and plans by minimizing the squared Euclidean distance between predicted and goal embeddings. In this work, we investigate the test-time robustness of this latent planning interface. Analyzing the learned metric on Push-T, we first find that task-relevant object state heavily dominates latent distance: shifting the block by a single pixel displaces the embedding 3 times further than an equivalent agent shift, and swapping the block pose alone accounts for 99% of the distance between scene pairs. However, because this pose information is distributed across a curved, high-dimensional manifold rather than a fixed linear subspace, the encoder cannot be trivially edited post-hoc to isolate or redirect it. We next demonstrate that this geometry is highly vulnerable at test time. A white-box infinity perturbation of just ε = 4/255 applied to a single input frame causes the encoder to register the block at an arbitrary, attacker-chosen pose in 73 -- 95% of scenes without disturbing the encoded agent state. Perturbing either the goal image or the live camera observation drops Push-T planning success from 87% to 32%, whereas uniform random noise at twice the magnitude has negligible impact. This vulnerability persists across all four standard LeWM benchmarks (Push-T, Two-Room, Reacher, and OGBench-Cube). At ε = 8/255, the attack bridges 82 -- 97% of the squared latent distance to the target embedding in every environment, inducing planning degradation proportional to each task's reliance on the spoofed entity.
Sources
- LeJEPA: Provable and Scalable Self-Supervised Learning Without the Heuristics
- On Evaluating Adversarial Robustness
- Reperesentation Geometry Matters for Planning with JEPA World Models
- LeWorldModel: Stable End-to-End Joint-Embedding Predictive Architecture from Pixels
- When the World Lies: Backdoor Attacks on Latent World Models for Downstream Control
- Learning from Reward-Free Offline Data: A Case for Planning with Latent Dynamics Models
- Exploring the Adversarial Vulnerabilities of Vision-Language-Action Models in Robotics
Related papers
- TEA: Text Encoder Alignment for Robust Concept Erasure in Text-to-Image Models
- Do Egocentric Video-Language Models Capture Both Hand- and Object-Centric Cues?
- Aura: Consistent Multi-Subject Video Generation via VLM-Grounded Semantic Alignment
- PVRF: All-in-one Adverse Weather Removal via Prior-modulated and Velocity-constrained Rectified Flow
- DLWM: Diverse Latent World Models for Efficient Multimodal Reasoning
- Out of Sight, Still in Mind: Token Compression for Omni-LLMs