BEE: Intervention-Adaptive Real-World Reinforcement Learning with Vision-Language-Action Models
cs.RO, cs.AI
Submitted: 2026-09-23
Updated: 2026-09-23
License: http://creativecommons.org/licenses/by/4.0/
The gist: Vision-language-action (VLA) models handle long-horizon manipulation, yet success hinges on a few precision-critical phases where millimeter-scale errors undo all prior progress.
Terminology
Abstract
Vision-language-action (VLA) models handle long-horizon manipulation, yet success hinges on a few precision-critical phases where millimeter-scale errors undo all prior progress. Online reinforcement learning (RL) can optimize exactly these actions, but free exploration is far too costly on real robots, which makes human corrections indispensable. However, existing online RL methods for VLAs either cannot incorporate such corrections or fold them into undifferentiated supervision. Yet human corrections are not uniformly noisy but reliable along some action dimensions and variable along others. Building on this, we introduce BEE, an intervention-adaptive framework for real-world RL on a frozen VLA that lets the policy go BEyond Expert imitation. We formulate human corrections not as actions to reproduce but as evidence about a constraint: a Correction Model predicts how a human would correct a given VLA proposal and how consistent the correction is along each action dimension. This predicted consistency sets the per-dimension tightness of a constraint on policy optimization. Where corrections are consistent the policy stays close to the human, and where they vary, the constraint relaxes. We evaluate BEE on three real-world manipulation tasks and one LIBERO-Pro simulation task at a matched online-data budget. BEE attains the highest success rate on every task, 91.2% on average against 57.5% for RLT and 42.1% for DSRL, and the lowest human intervention rate on all real-world tasks.
Sources
- RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Control
- $\pi_{0.5}$: a Vision-Language-Action Model with Open-World Generalization
- RL-100: Performant Robotic Manipulation with Real-World Reinforcement Learning
- Steering Your Diffusion Policy with Latent Space Reinforcement Learning
- RL Token: Bootstrapping Online RL with Vision-Language-Action Models
- Real-world Reinforcement Learning from Suboptimal Interventions
- VLA-RL: Towards Masterful and General Robotic Manipulation with Scalable Reinforcement Learning
- Interactive Post-Training for Vision-Language-Action Models
- GRAPE: Generalizing Robot Policy via Preference Alignment
- RLDG: Robotic Generalist Policy Distillation via Reinforcement Learning
- AWAC: Accelerating Online Reinforcement Learning with Offline Datasets
- Residual Policy Learning
- LIBERO-PRO: Towards Robust and Fair Evaluation of Vision-Language-Action Models Beyond Memorization
- LIBERO: Benchmarking Knowledge Transfer for Lifelong Robot Learning
Related papers
- FMT x: An Efficient and Asymptotically Optimal Extension of the Fast Marching Tree for Dynamic Replanning
- MPCFormer: A physics-informed data-driven approach for explainable socially-aware autonomous driving
- RoboLab: A High-Fidelity Simulation Benchmark for Analysis of Task Generalist Policies
- ARC: A Reasoning Recipe for Robot Foundation Models
- HRDexDB: A 4D Dexterous Grasping Dataset Across Human and Multiple Robot Embodiments
- APT: Action Expert Pretraining Improves Instruction Generalization of Vision-Language-Action Policies