SelFusion: Self-distillation for Diffusion Language Models
The episode discusses a paper titled "SelFusion: Self-distillation for Diffusion Language Models." The hosts explain that SelFusion proposes a new self-distillation framework for Diffusion Language Models (DLMs) by using their internal noising process as the learning source. They detail how it uses two modes—easy and hard—and a dynamic, bidirectional strategy to select the best teacher signal based on token accuracy, showing performance gains without needing an external teacher.