Influence-Directed Distillation: Solving the Diversity Bottleneck in Sampled-Token On-Policy Distillation
The discussion of Influence-Directed Distillation addresses a failure in Sampled-Token On-Policy Distillation where models lose diversity despite improving accuracy. The researchers propose using a diagnostic tool, First-Order Local Entropy Influence, to identify problematic updates. They then apply Divergence-Adaptive Shrinkage to selectively correct these entropy drains, resulting in highly efficient models that maintain both accuracy and broad solution variety.