KDFP: A first-principles approach to knowledge distillation in large language models
KDFP introduces Knowledge Distillation from First Principles for white-box general knowledge distillation in LLMs. It uses a pretrained student initialization and an upward-online projector with a 0.25 teacher/student ratio to distill outputs from attention, MLP, and residual streams at every block. This method outperforms existing techniques by improving performance across benchmarks while significantly increasing training efficiency.