MPC-Injection: Biasing Off-Policy Locomotion RL Toward Controller-Induced Behavior Basins
cs.RO
Submitted: 2026-06-24
Updated: 2026-09-24
Comments: 24 pages, 11 figures. CoRL 2026
Code: https://github.com/unitreerobotics/unitree_
License: http://creativecommons.org/licenses/by/4.0/
The gist: Reinforcement learning (RL) for locomotion frequently converges to locally optimal but undeployable behaviors, such as vibrating limbs or scooting on the torso, that maximize return without producing
Terminology
Abstract
Reinforcement learning (RL) for locomotion frequently converges to locally optimal but undeployable behaviors, such as vibrating limbs or scooting on the torso, that maximize return without producing a usable gait. We present MPC-Injection, a low-overhead method that steers RL toward a designer-preferred behavior by inserting transitions generated in the same environment by a model predictive controller (MPC). Unlike reward shaping, MPC-Injection does not require redesigning the task reward, and unlike adversarial imitation learning, it adds no discriminator, no kinematic retargeting, and no auxiliary objective. We analyze how the injected transitions bias the learning, allowing the policy to converge to behaviors that pure RL may fail to reach under simple reward functions. On a 2D walker in simulation and with sim-to-real evaluation on a Go2 quadruped, we show that MPC-Injection produces gaits qualitatively comparable to those of reward shaping and adversarial motion priors. We also show that MPC-Injection can complete a barrel roll that pure RL fails to achieve under the same simple reward and can select between trotting and bounding gaits only through changing the injected MPC data.
Sources
- Olaf: Bringing an Animated Character to Life in the Physical World
- Opening the Sim-to-Real Door for Humanoid Pixel-to-Action Policy Transfer
- AMP: Adversarial Motion Priors for Stylized Physics-Based Character Control
- Learning Agile Skills via Adversarial Imitation of Rough Partial Demonstrations
- ResMimic: From General Motion Tracking to Humanoid Whole-body Loco-Manipulation via Residual Learning
- TWIST: Teleoperated Whole-Body Imitation System
- Infusing model predictive control into meta-reinforcement learning for mobile robots in dynamic environments
- Jacta: A Versatile Planner for Learning Dexterous and Whole-body Manipulation
- Benchmarking Potential Based Rewards for Learning Humanoid Locomotion
- Learning to Run with Potential-Based Reward Shaping and Demonstrations from Video Data
- Lyapunov Design for Robust and Efficient Robotic Reinforcement Learning
- CBF-RL: Safety Filtering Reinforcement Learning in Training with Control Barrier Functions
- Concurrent Training of a Control Policy and a State Estimator for Dynamic and Robust Legged Locomotion
- Learning Quadrupedal Locomotion over Challenging Terrain
- Generative Predictive Control: Flow Matching Policies for Dynamic and Difficult-to-Demonstrate Tasks
- GMT: General Motion Tracking for Humanoid Whole-Body Control
- Leveraging Demonstrations for Deep Reinforcement Learning on Robotics Problems with Sparse Rewards
- Residual MPC: Blending Reinforcement Learning with GPU-Parallelized Model Predictive Control
- MPC-Net: A First Principles Guided Policy Search
- Differentiable MPC for End-to-end Planning and Control
Related papers
- FMT x: An Efficient and Asymptotically Optimal Extension of the Fast Marching Tree for Dynamic Replanning
- MPCFormer: A physics-informed data-driven approach for explainable socially-aware autonomous driving
- RoboLab: A High-Fidelity Simulation Benchmark for Analysis of Task Generalist Policies
- ARC: A Reasoning Recipe for Robot Foundation Models
- HRDexDB: A 4D Dexterous Grasping Dataset Across Human and Multiple Robot Embodiments
- APT: Action Expert Pretraining Improves Instruction Generalization of Vision-Language-Action Policies