RoboFollow: Unveiling the Instruction Following Mirage in Embodied Agents
cs.RO
Submitted: 2026-09-22
Updated: 2026-09-22
Code: https://github.com/AutoLab-SAI-SJTU/RoboFollow
License: http://creativecommons.org/licenses/by/4.0/
The gist: Modern embodied agents achieve impressive success rates, yet their actual instruction-following ability is far weaker than these numbers suggest.
Terminology
Abstract
Modern embodied agents achieve impressive success rates, yet their actual instruction-following ability is far weaker than these numbers suggest. We trace this illusion to a structural property we term low scene entropy: when a visual scene admits only one valid task, language becomes redundant and a policy can score highly while barely using it. We introduce RoboFollow, a diagnostic benchmark with three principles: (1) High Scene Entropy: each training scene supports multiple kinematically distinct task branches, making vision alone insufficient and forcing reliance on language. (2) Hierarchical Diagnostic Protocol: a four-level protocol (L0--L3) progressively perturbs visual layout and semantics, probing whether equivalent instructions yield consistent behavior and distinct ones yield discriminable behavior across spatial relations, attributes, trajectory constraints, and logic. (3) Confound-Controlled Diagnosis: we simplify interaction objects, restrict actions to the trained repertoire and report stage-wise Intent and Execution scores, isolating comprehension from motor execution. Evaluation of nine VLA and WAM policies shows that strong L0 performance, where attained, does not reliably transfer to L1--L3 under our fine-tuning setup. Representative mitigations, including stronger VLM backbones, QA co-training, LangForce, and Classifier-Free Guidance, all fail to close this gap. RoboFollow exposes genuine instruction following as a critical, overlooked bottleneck. Code and dataset are available at https://github.com/AutoLab-SAI-SJTU/RoboFollow and https://huggingface.co/datasets/AutoLab-SJTU/robofollow-data.
Sources
- OpenVLA: An Open-Source Vision-Language-Action Model
- $\pi_0$: A Vision-Language-Action Flow Model for General Robot Control
- Unleashing Large-Scale Video Generative Pre-training for Visual Robot Manipulation
- Motus: A Unified Latent Action World Model
- Multi-agent Embodied AI: Advances and Future Directions
- Seeing to Act, Prompting to Specify: A Bayesian Factorization of Vision Language Action Policy
- LIBERO-PRO: Towards Robust and Fair Evaluation of Vision-Language-Action Models Beyond Memorization
- LIBERO-Plus: In-depth Robustness Analysis of Vision-Language-Action Models
- GR00T N1: An Open Foundation Model for Generalist Humanoid Robots
- $\pi_{0.5}$: a Vision-Language-Action Model with Open-World Generalization
- VLM4VLA: Revisiting Vision-Language-Models in Vision-Language-Action Models
- Open X-Embodiment: Robotic Learning Datasets and RT-X Models
- RT-1: Robotics Transformer for Real-World Control at Scale
- Octo: An Open-Source Generalist Robot Policy
- X-VLA: Soft-Prompted Transformer as Scalable Cross-Embodiment Vision-Language-Action Model
- InternVLA-M1: A Spatially Guided Vision-Language-Action Framework for Generalist Robot Policy
- SpatialVLA: Exploring Spatial Representations for Visual-Language-Action Model
- TA-VLA: Elucidating the Design Space of Torque-aware Vision-Language-Action Models
- SmolVLA: A Vision-Language-Action Model for Affordable and Efficient Robotics
- QVLA: Not All Channels Are Equal in Vision-Language-Action Model's Quantization
Related papers
- FMT x: An Efficient and Asymptotically Optimal Extension of the Fast Marching Tree for Dynamic Replanning
- MPCFormer: A physics-informed data-driven approach for explainable socially-aware autonomous driving
- RoboLab: A High-Fidelity Simulation Benchmark for Analysis of Task Generalist Policies
- ARC: A Reasoning Recipe for Robot Foundation Models
- HRDexDB: A 4D Dexterous Grasping Dataset Across Human and Multiple Robot Embodiments
- APT: Action Expert Pretraining Improves Instruction Generalization of Vision-Language-Action Policies