Cross-Behavior Policy Optimization: Unified Learning from Offline Demonstrations and On- and Off-Policy Rollouts via a Shared Regression Target
CBPO unifies training from offline demonstrations and various rollouts (on-policy and off-policy) using a single regression loss driven by a fixed reference target. This allows disparate data sources to share a common format, eliminating the need for separate SFT and RL objectives or complex weighting schemes. The unified approach achieves leading performance across multiple models and benchmarks.