Find Something You Can't Do: Agentic Real-World Reinforcement Learning for Self-Improving VLA Models
cs.RO
Submitted: 2026-09-25
Updated: 2026-10-01
Code: https://github.com/huggingface/lerobot
License: http://creativecommons.org/licenses/by/4.0/
The gist: Vision--language--action (VLA) models provide strong priors for robotic manipulation but are typically deployed as frozen policies, unable to improve from their own failures.
Terminology
Abstract
Vision--language--action (VLA) models provide strong priors for robotic manipulation but are typically deployed as frozen policies, unable to improve from their own failures. Real-world reinforcement learning (RL) offers a path to continued improvement, yet manual environment resets and task-success supervision hinder autonomous learning. We introduce FIND, an agentic real-world RL framework that closes the loop between scene understanding, weakness-aware practice, self-evaluation, and policy improvement in a persistent workspace. FIND reframes autonomous practice as a scene-conditioned, performance-aware task-selection problem: instead of restoring a predefined scene after each rollout, it uses the resulting scene to determine what to practice next. A vision--language agent identifies feasible tasks from a predefined library, prioritizes those with lower recent success rates, and evaluates outcomes using paired pre- and post-execution observations. We instantiate FIND with a frozen π 0.5 VLA and residual off-policy RL. Across eight real-world manipulation tasks, the independent human-assessed success rate improves from 55% to 71.9%. A representative run completes 456 autonomous episodes within 6 hours of interaction, requiring 30 scene-recovery interventions and no human-provided reward labels during online learning. Ablations and systematic evaluations further examine key design choices, agent evaluation accuracy, and human intervention requirements. Our website is made publicly available at: FIND.github.io.
Sources
- RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Control
- $\pi_0$: A Vision-Language-Action Flow Model for General Robot Control
- $\pi_{0.5}$: a Vision-Language-Action Model with Open-World Generalization
- Self-Improving Robots: End-to-End Autonomous Visuomotor Reinforcement Learning
- ASPIRE: Agentic /Skills Discovery for Robotics
- Residual Off-Policy RL for Finetuning Behavior Cloning Policies
- RLDG: Robotic Generalist Policy Distillation via Reinforcement Learning
- RL Token: Bootstrapping Online RL with Vision-Language-Action Models
- When Life Gives You BC, Make Q-functions: Extracting Q-values from Behavior Cloning for On-Robot Reinforcement Learning
- Practice Makes Perfect: Planning to Learn Skill Parameter Policies
- Demonstration-Bootstrapped Autonomous Practicing via Multi-Task Reinforcement Learning
- Robot Trains Robot: Automatic Real-World Policy Adaptation and Learning for Humanoids
- Inner Monologue: Embodied Reasoning through Planning with Language Models
- PaLM-E: An Embodied Multimodal Language Model
- VoxPoser: Composable 3D Value Maps for Robotic Manipulation with Language Models
- Language to Rewards for Robotic Skill Synthesis
- Robots That Ask For Help: Uncertainty Alignment for Large Language Model Planners
- HARBOR: A Harness Framework for Agentic Robot Reinforcement Learning
- Nautilus: From One Prompt to Plug-and-Play Robot Learning
- Randomized Ensembled Double Q-Learning: Learning Fast Without a Model
Related papers
- FMT x: An Efficient and Asymptotically Optimal Extension of the Fast Marching Tree for Dynamic Replanning
- MPCFormer: A physics-informed data-driven approach for explainable socially-aware autonomous driving
- RoboLab: A High-Fidelity Simulation Benchmark for Analysis of Task Generalist Policies
- ARC: A Reasoning Recipe for Robot Foundation Models
- HRDexDB: A 4D Dexterous Grasping Dataset Across Human and Multiple Robot Embodiments
- APT: Action Expert Pretraining Improves Instruction Generalization of Vision-Language-Action Policies