Rethinking Relative Policy Differences Transfer of On-Policy Distillation
The paper introduces a unified view for policy optimization by defining an implicit reward based on log-probability ratios between proxy expert and reference policies. This enables 'any-to-any' transfer across different models, scales, or training stages. It shows that task-directed improvements transfer well, while model scaling alone does not.