Let the Batch Decide, On- or Off-Policy: Effective Sample Size for RL Post-Training
The P3O objective replaces fixed clipping with a batch-adaptive score-function cap and matching regularizer based on the effective sample size (ESS) of policy ratios. This allows the update to automatically tighten when data is stale or mismatched, behaving like an on-policy update on fresh data while retaining learning signals from older samples. It simplifies training by removing fixed hyperparameters.