Hide-and-Seek in Trajectories: Discovering Failure Signals for VLA Runtime Monitoring
cs.RO, cs.AI
Submitted: 2026-05-29
Updated: 2026-09-25
Comments: NeurIPS 2026
License: http://creativecommons.org/licenses/by/4.0/
The gist: Vision-Language-Action (VLA) models enable robots to follow natural language instructions and generalize across diverse tasks, but they remain vulnerable to execution failures that compromise
Terminology
Abstract
Vision-Language-Action (VLA) models enable robots to follow natural language instructions and generalize across diverse tasks, but they remain vulnerable to execution failures that compromise reliability in real-world deployment. Detecting such failures during execution is therefore critical for the robust deployment of embodied systems. Existing failure detection methods either rely on expensive action resampling or external models, while alternatives propagate trajectory-level labels uniformly across every timestep, obscuring localized failure signals. In this paper, we propose Hide-and-Seek, a framework that formulates VLA failure detection as a coarsely supervised learning problem. By combining inter-trajectory and intra-trajectory contrastive objectives, Hide-and-Seek localizes failure-indicative actions and induces temporally structured failure signals from trajectory-level supervision alone, without any step-level annotation. We evaluate Hide-and-Seek on LIBERO, VLABench, and a real-world robotic platform across three representative VLA policies: OpenVLA, π 0, and π 0.5.Our method achieves state-of-the-art multi-task failure detection performance with a practical accuracy--timeliness trade-off under conformal prediction, and generalizes well to both seen and unseen tasks.
Sources
- $\pi_0$: A Vision-Language-Action Flow Model for General Robot Control
- $\pi^{*}_{0.6}$: a VLA That Learns From Experience
- GR00T N1: An Open Foundation Model for Generalist Humanoid Robots
- VLANeXt Family: A Systematic Study of VLA Models from Core Recipes to Emerging Paradigms
- MEM: Multi-Scale Embodied Memory for Vision Language Action Models
- An Anatomy of Vision-Language-Action Models: From Modules to Milestones and Challenges
- EVE: A Generator-Verifier System for Generative Policies
- Grounding Multimodal LLMs to Embodied Agents that Ask for Help with Reinforcement Learning
- When to Act, Ask, or Learn: Uncertainty-Aware Policy Steering
- $\pi_{0.5}$: a Vision-Language-Action Model with Open-World Generalization
- RynnVLA-002: A Unified Vision-Language-Action and World Model
- FAST: Efficient Action Tokenization for Vision-Language-Action Models
- A Careful Examination of Large Behavior Models for Multitask Dexterous Manipulation
- World Action Models are Zero-shot Policies
- Weakly Supervised Anomaly Detection: A Survey
- The Importance of Being a Band: Finite-Sample Exact Distribution-Free Prediction Sets for Functional Data
- OpenAI GPT-5 System Card
- Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling
- Qwen3-VL Technical Report
- A Survey on Hallucination in Large Vision-Language Models
Related papers
- FMT x: An Efficient and Asymptotically Optimal Extension of the Fast Marching Tree for Dynamic Replanning
- MPCFormer: A physics-informed data-driven approach for explainable socially-aware autonomous driving
- RoboLab: A High-Fidelity Simulation Benchmark for Analysis of Task Generalist Policies
- ARC: A Reasoning Recipe for Robot Foundation Models
- HRDexDB: A 4D Dexterous Grasping Dataset Across Human and Multiple Robot Embodiments
- APT: Action Expert Pretraining Improves Instruction Generalization of Vision-Language-Action Policies