Disentangling Spurious Correlations in Vision-Language-Action Models via Predicting Domain-Invariant Latent Lookahead
cs.RO, cs.LG
Submitted: 2026-09-29
Updated: 2026-09-29
Comments: Accepted to CoRL 2026. Project website: https://dill-vla.github.io/
Project page: https://dill-vla.github.io
License: http://creativecommons.org/licenses/by/4.0/
The gist: Vision-Language-Action (VLA) models remain brittle under visual distribution shifts, often relying on spurious correlations tied to domain-specific factors rather than task-relevant structure.
Terminology
Abstract
Vision-Language-Action (VLA) models remain brittle under visual distribution shifts, often relying on spurious correlations tied to domain-specific factors rather than task-relevant structure. We propose Domain-Invariant Latent Lookahead (DILL), a representation-learning framework that mitigates shortcut learning in VLA policies. Our key idea is to supervise policies with domain-invariant future latents learned from domain-transformed trajectory data. A Task-Domain Encoder is trained with contrastive objectives and Gaussian disentanglement regularization to separate task-relevant structure from domain-specific visual variation. The learned encoder then provides future latents for VLA policy learning through lookahead prediction and domain disentanglement, encouraging the policy to focus on task-relevant structure rather than incidental visual factors. Counterfactual task-view evaluations show that DILL reduces shortcut reliance, while LIBERO-Plus evaluations demonstrate improved visual robustness, with 69.1% average success, 11.4 percentage points above the strongest baseline. Real-world manipulation experiments further support DILL's applicability beyond controlled simulation. Complementary latent-space diagnostics show that these behavioral gains are accompanied by representations that better preserve task-consistent structure while suppressing domain-specific variation. Our project page is available at https://dill-vla.github.io/.
Sources
- RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Control
- OpenVLA: An Open-Source Vision-Language-Action Model
- $\pi_0$: A Vision-Language-Action Flow Model for General Robot Control
- Open X-Embodiment: Robotic Learning Datasets and RT-X Models
- DROID: A Large-Scale In-The-Wild Robot Manipulation Dataset
- LIBERO-Plus: In-depth Robustness Analysis of Vision-Language-Action Models
- VLA-Arena: An Open-Source Framework for Benchmarking Vision-Language-Action Models
- LIBERO-X: Robustness Litmus for Vision-Language-Action Models
- Shortcut Learning in Generalist Robot Policies: The Role of Dataset Diversity and Fragmentation
- Qwen2.5-VL Technical Report
- FLARE: Robot Learning with Implicit World Modeling
- VLA-JEPA: Enhancing Vision-Language-Action Model with Latent World Model
- LIBERO-PRO: Towards Robust and Fair Evaluation of Vision-Language-Action Models Beyond Memorization
- View-Invariant Policy Learning via Zero-Shot Novel View Synthesis
- RoboUniView: Visual-Language Model with Unified View Representation for Robotic Manipulation
- InSpire: Vision-Language-Action Models with Intrinsic Spatial Reasoning
- Video Prediction Policy: A Generalist Robot Policy with Predictive Visual Representations
- Unleashing Large-Scale Video Generative Pre-training for Visual Robot Manipulation
- GR-2: A Generative Video-Language-Action Model with Web-Scale Knowledge for Robot Manipulation
- Ctrl-World: A Controllable Generative World Model for Robot Manipulation
Related papers
- FMT x: An Efficient and Asymptotically Optimal Extension of the Fast Marching Tree for Dynamic Replanning
- MPCFormer: A physics-informed data-driven approach for explainable socially-aware autonomous driving
- RoboLab: A High-Fidelity Simulation Benchmark for Analysis of Task Generalist Policies
- ARC: A Reasoning Recipe for Robot Foundation Models
- HRDexDB: A 4D Dexterous Grasping Dataset Across Human and Multiple Robot Embodiments
- APT: Action Expert Pretraining Improves Instruction Generalization of Vision-Language-Action Policies