RFPO: Rectified Flow Policy Optimization for Embodied Control
cs.RO
Submitted: 2026-10-07
Updated: 2026-10-07
Code: https://github.com/AIGeeksGroup/RFPO
Project page: https://aigeeksgroup.github.io/RFPO
License: http://creativecommons.org/licenses/by-nc-sa/4.0/
The gist: Flow-based policies provide an expressive framework for continuous robot control, but their iterative ODE integration incurs substantial inference cost.
Terminology
Abstract
Flow-based policies provide an expressive framework for continuous robot control, but their iterative ODE integration incurs substantial inference cost. Naively reducing the integration budget can severely degrade control, since policies optimized under full-step execution are not explicitly constrained to remain reliable under coarse numerical integration. We refer to this mismatch as the few-step discretization gap. To address this problem, we introduce RFPO, a flow-policy optimization framework for reliable few-step execution. Reward-aware online Reflow rectifies student-induced transport paths during on-policy learning, making the resulting policy more robust to coarse integration. A frozen Gaussian PPO controller supplies complementary action-space supervision at full and intermediate integration budgets, while the deployed policy remains a single flow student executed with one Euler step. Across Unitree Go2, Boston Dynamics Spot, Unitree H1, and Unitree G1, RFPO consistently preserves full-step control performance under one-step execution, with one-step returns remaining within 2.4% of their corresponding 64-step values across both zero and random initialization. On Unitree Go2, one-step execution retains 98.5% of the 64-step reward while reducing onboard mean inference latency from 4.39 ms to 0.08 ms, yielding a 54.9x speedup. Real-robot experiments further validate stable one-step locomotion. Code: https://github.com/AIGeeksGroup/RFPO. Website: https://aigeeksgroup.github.io/RFPO.
Sources
- $\pi_0$: A Vision-Language-Action Flow Model for General Robot Control
- DIME:Diffusion-Based Maximum Entropy Reinforcement Learning
- Flow Matching for Generative Modeling
- Flow Straight and Fast: Learning to Generate and Transfer Data with Rectified Flow
- Flow Q-Learning
- Consistency Policy: Accelerated Visuomotor Policies via Consistency Distillation
- Learning a Diffusion Model Policy from Rewards via Q-Score Matching
- Proximal Policy Optimization Algorithms
- Flow Policy Gradients for Robot Control
- 3D Diffusion Policy: Generalizable Visuomotor Policy Learning via Simple 3D Representations
Related papers
- FMT x: An Efficient and Asymptotically Optimal Extension of the Fast Marching Tree for Dynamic Replanning
- MPCFormer: A physics-informed data-driven approach for explainable socially-aware autonomous driving
- RoboLab: A High-Fidelity Simulation Benchmark for Analysis of Task Generalist Policies
- ARC: A Reasoning Recipe for Robot Foundation Models
- HRDexDB: A 4D Dexterous Grasping Dataset Across Human and Multiple Robot Embodiments
- APT: Action Expert Pretraining Improves Instruction Generalization of Vision-Language-Action Policies