When Vision Overrides Language: Evaluating and Mitigating Counterfactual Failures in VLAs
cs.CV, cs.RO
Submitted: 2026-02-19
Updated: 2026-11-04
Comments: Accepted to CoRL 2026 (Spotlight). Website: https://vla-cf.github.io/
Project page: https://vla-cf.github.io
License: http://creativecommons.org/licenses/by/4.0/
The gist: Vision-Language-Action models (VLAs) promise to ground language instructions in robot control, yet in practice often fail to follow language faithfully.
Terminology
Abstract
Vision-Language-Action models (VLAs) promise to ground language instructions in robot control, yet in practice often fail to follow language faithfully. When presented with instructions that lack strong scene-specific supervision, VLAs suffer from counterfactual failures: they act based on vision shortcuts induced by dataset biases, repeatedly executing well-learned behaviors and selecting objects frequently seen during training regardless of language intent. To support systematic study, we introduce LIBERO-CF, the first counterfactual benchmark for VLAs that evaluates language following capability by assigning alternative, feasible task instructions under the same LIBERO scene layouts. Our evaluation reveals that counterfactual failures are prevalent yet underexplored across state-of-the-art VLAs. To mitigate these failures, we propose Counterfactual Action Guidance (CAG), a simple yet effective dual-branch inference scheme that explicitly strengthens language conditioning in VLAs. CAG combines a standard VLA policy with a parallel language-unconditioned branch, amplifying the difference between their action predictions induced by language instructions. This design reduces reliance on vision shortcuts, improves robustness on counterfactual tasks, and requires neither additional demonstrations nor modifications to the base VLA architecture or its pretrained weights. Extensive experiments demonstrate consistent improvements across diverse VLAs. On LIBERO-CF, our training-free variant, TF-CAG, improves π 0.5 by 9.7% in language grounding and 3.6% in task success. When using a separate Vision-Action (VA) module, VA-CAG achieves larger gains of 15.5% and 8.5%, respectively. In real-world evaluations, VA-CAG improves language grounding by 9.4% and task success by 17.2% on average.
Sources
- GPT-4 Technical Report
- PaliGemma: A versatile 3B VLM for transfer
- GR00T N1: An Open Foundation Model for Generalist Humanoid Robots
- $\pi_0$: A Vision-Language-Action Flow Model for General Robot Control
- RT-1: Robotics Transformer for Real-World Control at Scale
- RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Control
- AgiBot World Colosseo: A Large-scale Manipulation Platform for Scalable and Intelligent Embodied Systems
- RynnVLA-002: A Unified Vision-Language-Action and World Model
- WorldVLA: Towards Autoregressive Action World Model
- GR-3 Technical Report
- Moto: Latent Motion Token as the Bridging Language for Learning Robot Manipulation from Videos
- Diffusion Policy: Visuomotor Policy Learning via Action Diffusion
- OpenHelix: A Short Survey, Empirical Analysis, and Open-Source Dual-System VLA Model for Robotic Manipulation
- Interleave-VLA: Enhancing Robot Manipulation with Interleaved Image-Text Instructions
- LIBERO-Plus: In-depth Robustness Analysis of Vision-Language-Action Models
- CAST: Counterfactual Labels Improve Instruction Following in Vision-Language-Action Models
- Classifier-Free Diffusion Guidance
- $\pi_{0.5}$: a Vision-Language-Action Model with Open-World Generalization
- DreamGen: Unlocking Generalization in Robot Learning through Video World Models
- Mixture of Horizons in Action Chunking
Related papers
- TEA: Text Encoder Alignment for Robust Concept Erasure in Text-to-Image Models
- Do Egocentric Video-Language Models Capture Both Hand- and Object-Centric Cues?
- Aura: Consistent Multi-Subject Video Generation via VLM-Grounded Semantic Alignment
- PVRF: All-in-one Adverse Weather Removal via Prior-modulated and Velocity-constrained Rectified Flow
- DLWM: Diverse Latent World Models for Efficient Multimodal Reasoning
- Out of Sight, Still in Mind: Token Compression for Omni-LLMs