Token Utility for On-Policy Distillation via Local Trust-Region Optimization
The authors propose deriving token importance for on-policy distillation (OPD) directly from the optimization objective rather than using heuristic signals like uncertainty or disagreement. They define a utility based on how much each position's loss can decrease under an equal local KL budget, leading to Fisher Utility Weighting (FUW). This method outperforms uniform weighting across seven benchmarks and three models for both reverse-KL and forward-KL distillation.