Revisiting the Capacity Gap in Chain-of-Thought Distillation from a Practical Perspective

Name: Revisiting the Capacity Gap in Chain-of-Thought Distillation from a Practical Perspective
Uploaded: 2026-10-03T06:09:18.280150+00:00
Duration: 880 s
Description: The paper re-examines how capacity gaps affect Chain-of-Thought (CoT) distillation by fixing flawed experimental methods. It found that teacher strength often outweighs capacity gap issues, provided evaluation protocols are improved. The authors propose a new testing framework to give practitioners better guidance on selecting teacher and student models for practical deployment.

The paper re-examines how capacity gaps affect Chain-of-Thought (CoT) distillation by fixing flawed experimental methods. It found that teacher strength often outweighs capacity gap issues, provided evaluation protocols are improved. The authors propose a new testing framework to give practitioners better guidance on selecting teacher and student models for practical deployment.

Open the full episode