Dissecting Advantage-Guided Post-Training for Vision-Language-Action Policies
cs.RO
Submitted: 2026-09-23
Updated: 2026-09-23
Comments: 9 pages, 5 figures
Code: https://github.com/huggingface/lerobot
License: http://creativecommons.org/licenses/by-nc-nd/4.0/
The gist: Advantage-guided reinforcement learning provides a practical way to post-train vision-language-action (VLA) policies using limited robot data.
Terminology
Abstract
Advantage-guided reinforcement learning provides a practical way to post-train vision-language-action (VLA) policies using limited robot data. However, its performance depends on several coupled choices, including how critic-derived advantages are constructed, calibrated, and used for policy training. Existing recipes often combine these choices into a single end-to-end procedure, making their individual effects difficult to identify. In this work, we dissect advantage-guided VLA post-training through a controlled empirical study that separates these design choices while accounting for their distinct estimands. We develop stage-specific offline evaluation methods to screen alternative choices efficiently, without requiring extensive real-robot policy evaluations for every possible combination. The staged evaluation identifies a modular recipe that combines temporal-difference advantage construction, group-wise calibration, and continuous advantage weighting. Across four real-world bimanual tasks, the resulting recipe improves mean task progress and success over the SFT initialization by 0.42 and 0.63, respectively. Moreover, the proposed evaluation diagnostics show an overall alignment with downstream real-world performance, supporting their use for interpreting empirical outcomes and selecting advantage-guided post-training designs in practice.
Sources
- RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Control
- OpenVLA: An Open-Source Vision-Language-Action Model
- $\pi_{0.5}$: a Vision-Language-Action Model with Open-World Generalization
- Interactive Post-Training for Vision-Language-Action Models
- TGRPO :Fine-tuning Vision-Language-Action Model via Trajectory-wise Group Relative Policy Optimization
- ALOE: Action-Level Off-Policy Evaluation for Vision-Language-Action Model Post-Training
- RFTF: Reinforcement Fine-tuning for Embodied Agents with Temporal Feedback
- Using Non-Expert Data to Robustify Imitation Learning via Offline Reinforcement Learning
- CO-RFT: Efficient Fine-Tuning of Vision-Language-Action Models through Chunked Offline Reinforcement Learning
- ARM: Advantage Reward Modeling for Long-Horizon Manipulation
- Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learning
- AWAC: Accelerating Online Reinforcement Learning with Offline Datasets
- $\pi^{*}_{0.6}$: a VLA That Learns From Experience
- Proximal Policy Optimization Algorithms
- Stratified GRPO: Handling Structural Heterogeneity in Reinforcement Learning of LLM Search Agents
- SmolVLM: Redefining small and efficient multimodal models
- Hierarchical Advantage Weighting for Online RL Fine-Tuning of VLAs from Sparse Episode Outcomes
- Learning While Deploying: Fleet-Scale Reinforcement Learning for Generalist Robot Policies
Related papers
- FMT x: An Efficient and Asymptotically Optimal Extension of the Fast Marching Tree for Dynamic Replanning
- MPCFormer: A physics-informed data-driven approach for explainable socially-aware autonomous driving
- RoboLab: A High-Fidelity Simulation Benchmark for Analysis of Task Generalist Policies
- ARC: A Reasoning Recipe for Robot Foundation Models
- HRDexDB: A 4D Dexterous Grasping Dataset Across Human and Multiple Robot Embodiments
- APT: Action Expert Pretraining Improves Instruction Generalization of Vision-Language-Action Policies