Sandwich-Residuals: Parameter-Efficient Test-time Adaptation of World Models
cs.RO
Submitted: 2026-09-18
Updated: 2026-09-18
Project page: https://sandwich-residuals.github.io
License: http://creativecommons.org/licenses/by/4.0/
The gist: Latent world models enable planning by predicting the effects of actions in a learned representation space, but their predictions can become unreliable when test-time conditions differ from training.
Terminology
Abstract
Latent world models enable planning by predicting the effects of actions in a learned representation space, but their predictions can become unreliable when test-time conditions differ from training. Existing test-time adaptation methods address this by updating parts of the pretrained model, often modifying millions of parameters and requiring a choice of which internal components to adapt. We introduce Sandwich-Residuals, a lightweight alternative that keeps the pretrained world model frozen and learns only small residual corrections around the predictor. The residuals are optimized online using the model's self-supervised prediction error and require no rewards, labels, or source-domain data. Across 21 conditions on the AdaJEPA benchmark, our method achieves 1.3 times the success rate of the frozen model while retaining 95% of the performance of the strongest AdaJEPA variant and adapting 97-99% fewer parameters. Under compound shifts, this advantage increases to 1.9 times the success rate of the frozen model, while remaining comparable to internal block adaptation. We further demonstrate the same adaptation principle on a DINO-WM model for 3-D manipulation. These results suggest that effective test-time adaptation of world models does not necessarily require modifying their pretrained internal weights.
Sources
- Joint Embedding Predictive Architectures Focus on Slow Features
- Temporal Straightening for Latent Planning
- LeWorldModel: Stable End-to-End Joint-Embedding Predictive Architecture from Pixels
- A System-Level View on Out-of-Distribution Data in Robotics
- AdaJEPA: An Adaptive Latent World Model
- JEPA-Bisim: Learning Robust Visual Representations for Planning with Joint-Embedding Predictive World Models
- DWM: Separating World Effects from Actions in Latent World Models
Related papers
- FMT x: An Efficient and Asymptotically Optimal Extension of the Fast Marching Tree for Dynamic Replanning
- MPCFormer: A physics-informed data-driven approach for explainable socially-aware autonomous driving
- RoboLab: A High-Fidelity Simulation Benchmark for Analysis of Task Generalist Policies
- ARC: A Reasoning Recipe for Robot Foundation Models
- HRDexDB: A 4D Dexterous Grasping Dataset Across Human and Multiple Robot Embodiments
- APT: Action Expert Pretraining Improves Instruction Generalization of Vision-Language-Action Policies