Spoofing the Latent State: Imperceptible Goal-Image Attacks Derail Planning in LeWorldModel

arXiv:2611.00427 · cs.CV, cs.CR, cs.LG · Submitted 2026-10-30 · Read on arXiv

cs.CV, cs.CR, cs.LG

Submitted: 2026-10-30

Updated: 2026-10-30

Comments: 14 pages, 5 figures, 13 tables

License: http://creativecommons.org/licenses/by-sa/4.0/

The gist: Joint-embedding predictive architectures (JEPAs) have recently emerged as general-purpose visual world models for control.

Terminology

Abstract

Joint-embedding predictive architectures (JEPAs) have recently emerged as general-purpose visual world models for control. A prominent example is LeWorldModel (LeWM), which trains end-to-end from pixels using an isotropic Gaussian anti-collapse regularizer (SIGReg) and plans by minimizing the squared Euclidean distance between predicted and goal embeddings. In this work, we investigate the test-time robustness of this latent planning interface. Analyzing the learned metric on Push-T, we first find that task-relevant object state heavily dominates latent distance: shifting the block by a single pixel displaces the embedding 3 times further than an equivalent agent shift, and swapping the block pose alone accounts for 99% of the distance between scene pairs. However, because this pose information is distributed across a curved, high-dimensional manifold rather than a fixed linear subspace, the encoder cannot be trivially edited post-hoc to isolate or redirect it. We next demonstrate that this geometry is highly vulnerable at test time. A white-box infinity perturbation of just ε = 4/255 applied to a single input frame causes the encoder to register the block at an arbitrary, attacker-chosen pose in 73 -- 95% of scenes without disturbing the encoded agent state. Perturbing either the goal image or the live camera observation drops Push-T planning success from 87% to 32%, whereas uniform random noise at twice the magnitude has negligible impact. This vulnerability persists across all four standard LeWM benchmarks (Push-T, Two-Room, Reacher, and OGBench-Cube). At ε = 8/255, the attack bridges 82 -- 97% of the squared latent distance to the target embedding in every environment, inducing planning degradation proportional to each task's reliance on the spoofed entity.

Sources

Related papers