On-Policy Distillation with Negative-Policy Rollouts
Negative-Policy Distillation (NP-OPD) enhances On-Policy Distillation by adding rollouts from a lower-performing negative policy to provide an explicit negative signal. This complements teacher supervision, helping the student model avoid tokens preferred by the negative policy while maintaining positive guidance from the teacher. Experiments show this improves performance across various tasks and scales.