Two mathematical models of knowledge distillation
Listen
Radio episode about this paper
Transcript
Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.
Tom: I'm Tom.
Jane: And I'm Jane, here with Lu, Meng and our in-house large language model, Lalam.
Tom: We're looking at the paper: "Two mathematical models of knowledge distillation".
Jane: Here's the gist of the paper. Many hypotheses compete to explain the successes of knowledge distillation by proposing that distillation’s performance stems from easier optimization landscapes The paper proposes and analyzes a mathematical model of distillation suggesting that performance comes not from obtaining better models but from easier to optimize landscapes In generalized linear models, the authors prove that distillation fits performant student models asymptotically more quickly than non-distilled models In rank-1 matrix approximation, they characterize conditions on the target matrix under which gradient descent with distillation converges strictly faster than training on the supervised objective Theoretical Frameworks The paper formalizes knowledge distillation for maximum likelihood estimation in generalized linear models and low-rank matrix approximation For generalized linear models, the teacher generates distilled labels via a mapping h: T × X → Y′ representing the prediction on input x ∈ X The student model then minimizes an objective function LS: S → R which typically aggregates per-sample losses on the domain S ×V ×(Y ∪Y′) over the dataset In rank-r matrix approximation, the teacher LT (X) and student LS(x) objective functions are defined, and in distillation, the student objective LS has critical points at ±√λivi for i ∈
n: (or i ∈
r: ), depending on the target Online Optimization Dynamics In the online setting for generalized linear models, the SGD update rule is given by γt = γt−1 − ηt∇lS(γt−1; vt, y˜t) The authors show that both supervised learning and knowledge distillation converge to the same limiting parameter γ∗:= argminγ LS(γ) However, the SGD iterates obtained from knowledge distillation, β¯t, are asymptotically more efficient than those from supervised learning, ¯αt This efficiency gain is most pronounced when the response has high variance Finite Dataset Convergence When considering a fixed dataset D = (xi, yi)n i=1 ∼ Pn, the averaged iterates ¯γt are consistent and asymptotically normal Theorem 3.4 shows that both supervised learning ¯αt and knowledge distillation β¯t tend to the same minimizer of the empirical loss Furthermore, it is shown that for sufficiently large n, Cn(ˆγ) − C′n(ˆγ) ⪰ 0 with high probability Rank-r Matrix Approximation In the rank-r matrix approximation setting, both gradient descent applied to either the target or the teacher approximation converge to the same set of global optima Distillation converges faster under the sufficient condition that the matrix dimension scales inversely with the target error tolerance, raised to a spectrum-dependent constant Lemma 4.1.2 shows that knowledge distillation can improve this rate unless the teacher itself has rank r = 1 Experimental Validation Experiments on synthetic data, regression with FashionMNIST, and GPT-2 trained on C4 corroborate the theoretical predictions For generalized linear models, distillation improves efficiency as label variance and student dimension increase In rank-r matrix approximation, distillation accelerates convergence when the teacher rank approaches the student’s rank and when it truncates relatively large eigenvalues The results on language model pretraining show that distillation consistently converges in fewer steps across all student dimensions The work concludes that for generalized linear models, distillation produces asymptotically more efficient SGD iterates in both online and finite-dataset settings In rank-r matrix approximation, the authors prove that both approaches converge to the same set of global optima and identify conditions under which distillation converges strictly faster The theoretical analysis is corroborated by experiments on synthetic data, regression on real data, and language model pretraining The limitations suggest future work should extend the analysis to more complex optimization landscapes Acknowledgments This work was partially supported by the Office for Naval Research under grant N00014-22-1-2669, NSF AI Institute for Foundations of Machine Learning (IFML), and Open Philanthropy The authors would also like to thank Suhas Kotha and Gregory Xie for helpful feedback on the paper References Jimmy Ba and Rich Caruana.
Tom: Do deep nets really need to be deep?
Jane: In Zoubin Ghahramani, Max Welling, Corinna Cortes, Neil D.
Tom: Lawrence, and Kilian Q.
Title and authors: Tom: So let's talk about the title of this paper, "Two mathematical models of knowledge distillation," and who actually wrote it. It’s by Audrey Xie, Ludwig Schmidt, and John Duchi from Stanford University.
Jane: That’s a big team coming to this problem, so I'm curious how they frame it—why two models? What is the main idea behind that specific framing?
Lu: The core idea is that knowledge distillation isn't just one thing; it behaves differently depending on the mathematical setting you are in, so they have to look at different frameworks.
Meng: So, we aren't talking about a single universal distillation technique here; we’re looking at two specific mathematical scenarios where the benefits manifest differently.
Tom: Right, so one model handles things like generalized linear models, and the other deals with rank-r matrix approximation. That shows they are not just applying a trick everywhere.
Jane: And that distinction is key because it allows them to prove different kinds of convergence guarantees for each setting.
Lu: In the generalized linear model context, they formalize distillation for maximum likelihood estimation, which involves mapping teacher predictions onto the student's loss function.
Meng: That sounds like a lot of notation. For me, I just want to know what that means for deployment—is it a standard technique or something more specialized?
Tom: It’s specialized in its analysis, but the underlying concept is transferring predictive ability from a larger teacher model to a smaller student model by training the student on teacher-generated labels.
Jane: So, if we translate that jargon, they are essentially saying that distillation works because it changes the objective function for the student model in a way that makes finding an optimal solution easier.
Lu: They show how to define these objectives for both generalized linear models and low-rank matrix approximation, setting up the stage for their deeper mathematical proofs.
Meng: So, they are building two separate mathematical proofs instead of one massive proof covering everything at once. That’s a practical approach.
Tom: Precisely, because the mathematics required to rigorously prove convergence in both settings is quite different, so they tackle them separately to establish solid ground for each case.
Jane: So, even though it's two models, they are building toward a clearer understanding of what makes distillation successful across different types of machine learning problems.
The paper's summary: Tom: Okay, so what is the actual takeaway from this paper? They summarize by proposing that distillation’s success stems not from getting better models, but from easier to optimize landscapes.
Jane: That’s a powerful shift in perspective—it moves the focus away from just model quality and onto the training dynamics themselves.
Lu: They use mathematical modeling to show that distillation fits performant student models asymptotically more quickly than non-distilled models in generalized linear models.
Meng: So, when you say "asymptotically more quickly," what does that look like for a practical engineer? Is it a guaranteed speedup every single time?
Tom: It’s not guaranteed every single time, but over the long run, when you look at the whole training process, distillation is asymptotically faster.
Jane: And they do this by formalizing the loss function using a convex combination of losses between ground truth labels and teacher predictions.
Lu: In rank-r matrix approximation, they characterize specific conditions on the target matrix under which gradient descent with distillation converges strictly faster than training on the standard supervised objective.
Meng: So, in short, for some problems, you can trade a bit of training time for a guaranteed faster convergence rate when you use distillation.
Tom: That's right—it’s about optimization speed, not generalization performance itself; it helps delineate those two things clearly.
Jane: So the main point is that we have a mathematical model suggesting that the benefit comes from making the landscape easier to navigate during training.
The paper's improvements: Lu: Moving on to their proposed improvements, they highlight several specific advantages based on their theoretical findings.
Tom: They suggest that for generalized linear models, distillation improves efficiency when label variance and student dimension increase, which is a key finding in that setting.
Jane: So if I have high variance in my labels, using distillation seems to be even more beneficial when I'm training a larger student model?
Lu: Yes, the paper shows that it reduces the asymptotic covariance of the gradient noise compared to supervised learning, which is most pronounced when the response has high variance.
Meng: That’s interesting because in production, high variance usually means your data is messy; if distillation helps tame that noise in gradients, that's valuable.
Tom: And for low-rank matrix approximation, they prove that distillation converges faster under the sufficient condition where the matrix dimension scales inversely with the target error tolerance, raised to a spectrum-dependent constant.
Jane: So this suggests there are specific structural requirements on your target matrix and its dimensions that make distillation shine in those scenarios.
Lu: And they also point out a case where it can improve the rate of convergence unless the teacher itself has rank one, which is an important boundary condition to keep in mind.
Meng: That means if my teacher model is already very simple, say rank one, the benefit disappears for that specific problem. That’s a good caveat for scaling my models up.
Tom: The experimental validation on things like regression with FashionMNIST and GPT-two trained on C4 confirms these theoretical predictions across different data types <ref:2611.17230#pg2,and GPT-2 trained on C4>.
Jane: So overall, it seems the paper gives us concrete conditions—some are about label variance, some are about matrix structure, that tell us when to expect a performance gain from distillation.
Conclusion: Tom: So to wrap up, the main conclusion is that for generalized linear models, distillation produces asymptotically more efficient stochastic gradient descent iterates in both online and finite-dataset settings.
Jane: And they also proved that in rank-r matrix approximation, both approaches converge to the same set of global optima and identified conditions under which distillation converges strictly faster.
Lu: The paper successfully models the theoretical framework for maximum likelihood estimation in generalized linear models and low-rank matrix approximation, showing how distillation fits performant student models asymptotically more quickly than non-distilled ones.
Meng: From a production standpoint, this means that if you are dealing with high variance data or if you’re training very large models, the efficiency gains from distillation might be worth investigating.
Tom: It’s about understanding the optimization dynamics—it's not just about getting a better final model; it's about how fast we get there.
Jane: This paper provides a solid theoretical backing for why we should trust the distillation method in certain settings, provided you understand the conditions they laid out.
Lu: And their work sets up future research by suggesting extensions into more complex optimization landscapes to see how these ideas play out beyond these initial models.
Meng: I'm just hoping that when we look at real traffic, this theoretical efficiency translates into something tangible for us on the ground, and that’s the challenge.
Stanford University
cs.LG
Submitted: 2026-11-12
Updated: 2026-11-12
Code: https://github.com/0aax/two-models-kd
License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
Importance score: 94/100
The gist: Many hypotheses compete to explain the successes of knowledge distillation by proposing that distillation’s performance stems from easier optimization landscapes The paper proposes and analyzes a…
Key concepts
- Generalized Linear Models
- This framework formalizes knowledge distillation for maximum likelihood estimation. The teacher generates distilled labels based on a mapping from input data to predicted outputs, which the student model then minimizes an objective function over.
- Online Optimization Dynamics
- When training online using Stochastic Gradient Descent (SGD), the authors show that both supervised learning and knowledge distillation converge to the same final parameter. However, the iterates generated by distillation are asymptotically more efficient than those from standard supervised learning, especially when the response has high variance.
- Rank-r Matrix Approximation
- This setting involves approximating a target matrix using a lower-rank matrix. The paper analyzes conditions where knowledge distillation accelerates gradient descent convergence compared to training directly on the supervised objective, particularly when the teacher's rank is relevant.
Terminology
Summary
Many hypotheses compete to explain the successes of knowledge distillation by proposing that distillation’s performance stems from easier optimization landscapes The paper proposes and analyzes a mathematical model of distillation suggesting that performance comes not from obtaining better models but from easier to optimize landscapes In generalized linear models, the authors prove that distillation fits performant student models asymptotically more quickly than non-distilled models In rank-1 matrix approximation, they characterize conditions on the target matrix under which gradient descent with distillation converges strictly faster than training on the supervised objective
Theoretical Frameworks
The paper formalizes knowledge distillation for maximum likelihood estimation in generalized linear models and low-rank matrix approximation For generalized linear models, the teacher generates distilled labels via a mapping h: T × X → Y′ representing the prediction on input x ∈ X The student model then minimizes an objective function LS: S → R which typically aggregates per-sample losses on the domain S ×V ×(Y ∪Y′) over the dataset In rank-r matrix approximation, the teacher LT (X) and student LS(x) objective functions are defined, and in distillation, the student objective LS has critical points at ±√λivi for i ∈ [n] (or i ∈ [r]), depending on the target
Online Optimization Dynamics
In the online setting for generalized linear models, the SGD update rule is given by γt = γt−1 − ηt∇lS(γt−1; vt, y˜t) The authors show that both supervised learning and knowledge distillation converge to the same limiting parameter γ∗:= argminγ LS(γ) However, the SGD iterates obtained from knowledge distillation, β¯t, are asymptotically more efficient than those from supervised learning, ¯αt This efficiency gain is most pronounced when the response has high variance
Finite Dataset Convergence
When considering a fixed dataset D = (xi, yi)n i=1 ∼ Pn, the averaged iterates ¯γt are consistent and asymptotically normal Theorem 3.4 shows that both supervised learning ¯αt and knowledge distillation β¯t tend to the same minimizer of the empirical loss Furthermore, it is shown that for sufficiently large n, Cn(ˆγ) − C′n(ˆγ) ⪰ 0 with high probability
Rank-r Matrix Approximation
In the rank-r matrix approximation setting, both gradient descent applied to either the target or the teacher approximation converge to the same set of global optima Distillation converges faster under the sufficient condition that the matrix dimension scales inversely with the target error tolerance, raised to a spectrum-dependent constant Lemma 4.1.2 shows that knowledge distillation can improve this rate unless the teacher itself has rank r = 1
Experimental Validation
Experiments on synthetic data, regression with FashionMNIST, and GPT-2 trained on C4 corroborate the theoretical predictions For generalized linear models, distillation improves efficiency as label variance and student dimension increase In rank-r matrix approximation, distillation accelerates convergence when the teacher rank approaches the student’s rank and when it truncates relatively large eigenvalues The results on language model pretraining show that distillation consistently converges in fewer steps across all student dimensions
Conclusion
The work concludes that for generalized linear models, distillation produces asymptotically more efficient SGD iterates in both online and finite-dataset settings In rank-r matrix approximation, the authors prove that both approaches converge to the same set of global optima and identify conditions under which distillation converges strictly faster The theoretical analysis is corroborated by experiments on synthetic data, regression on real data, and language model pretraining The limitations suggest future work should extend the analysis to more complex optimization landscapes
Acknowledgments
This work was partially supported by the Office for Naval Research under grant N00014-22-1-2669, NSF AI Institute for Foundations of Machine Learning (IFML), and Open Philanthropy The authors would also like to thank Suhas Kotha and Gregory Xie for helpful feedback on the paper
References
Jimmy Ba and Rich Caruana. Do deep nets really need to be deep? p. 8 In Zoubin Ghahramani, Max Welling, Corinna Cortes, Neil D. p. 8 Lawrence, and Kilian Q. p. 8 Weinberger, editors, Advances in Neural Information Processing Systems 27: Annual Conference on Neural Information Processing Systems 2014 Lucas Beyer et al. p. 8 Knowledge distillation: A good teacher is patient and consistent In IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2022 Samuel Burer and Renato D. p. 8 C. Monteiro. A nonlinear programming algorithm for solving semidefinite programs via low-rank factorization Math. p. 8 Program., 95(2):329–357, 2003 Dan Busbridge et al. p. 8 Distillation scaling laws In Aarti Singh, Maryam Fazel, Daniel Hsu, Simon Lacoste-Julien, Felix Berkenkamp, Tegan Maharaj, Kiri Wagstaff, and Jerry Zhu editors Forty-second International Conference on Machine Learning ICML 2025 Tri Dao et al. p. 8 Knowledge distillation as semiparametric inference In 9th International Conference on Learning Representations ICLR 2021 Rudrajit Das and Sujay Sanghavi. p. 8 Understanding selfdistillation in the presence of label noise In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett editors International Conference on Machine Learning ICML 2023 PMLR 2023. p. 8 URL https://proceedings. p. 8mlr.press/v202/das23d. p. 8html
Appendix A Related Work Continued
Knowledge distillation has shown repeated success in training small performant models using larger, more capable models Menon et al. (2020) connects teacher predictions in classification to modeling the Bayes class probabilities Dao et al. p. 0 (2021) views distillation as a plug-in method for semiparameteric inference, where the teacher predictions serve as a plug-in estimate of the Bayes class probabilities Mobahi et al. p. 8 (2020) analyze self-distillation, where teacher and student are the same, in a Hilbert space with l2 regularization Das and Sanghavi (2023) provide another explanation for self-distillation Pareek et al. p. 9 (2024) generalizes this to multistep self-distillation Ildiz et al. p. 0 (2025) analyze knowledge distillation in high-dimensional ridgeless linear regression with Gaussian covariates
Detailed Assumptions and Deferred Proofs for Section 3.1
Assumption B.0.1 states that the per-sample loss lS(γ; ˜z) and population loss LeS(γ) satisfy conditions regarding the Hessian at the unique optimal point γ∗ Corollary B. p. 40.1 states that under Assumptions B. p. 30.1 and 3.0.2, the averaged SGD iterates are asymptotically normal with a specific second moment of gradient noise
Detailed Assumptions and Deferred Proofs for Section 3.2
Assumption C.0.1 formalizes the assumptions for the empirical loss Len S(γ) and states that under these conditions, the averaged iterates are asymptotically normal with an empirical second moment of gradient noise
Detailed Assumptions and Deferred Proofs for Section 4
The analysis in Section 4 shows that the set of all critical points is defined as ±√λiei n i=1 ∪ ±√λiei r i=1 The proof of Theorem 4.
Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.
Improvements for AI systems
-
Faster optimization for Generalized Linear Models by training on teacher predictions instead of ground-truth labels leads to asymptotically more efficient Stochastic Gradient Descent iterates, with an efficiency gap proportional to the label variance (Theorem 3.2).
-
Improved convergence speed in low rank matrix approximation by using knowledge distillation targets under sufficient conditions, as gradient descent converges strictly faster under these conditions (Theorem 4.2).
-
Enhanced performance in high-dimensional regression tasks by leveraging distillation on real data, where the student achieves lower parameter error with a larger initial step size when the student dimension is large (Figure 6).
-
Accelerated training of language models during pretraining by minimizing KL-divergence between teacher and student distributions instead of cross-entropy with ground truth labels, leading to fewer training steps across subsets of increasing entropy (Figure 3).
-
More robust estimation in high variance scenarios by demonstrating that knowledge distillation reduces the asymptotic covariance of the gradient noise compared to supervised learning, which is most pronounced when the response has high variance (Theorem 3.2).
Sources
- Distilling the Knowledge in a Neural Network
- Training Compute-Optimal Large Language Models
- Why distillation helps: a statistical perspective
- Gemma 2: Improving Open Language Models at a Practical Size
- FitNets: Hints for Thin Deep Nets
- DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter
- LLM Pruning and Distillation in Practice: The Minitron Approach
- Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms
Related papers
- A Physics-Regulated Neural Framework for Learning 3D Grain Growth Dynamics
- FlexMoE: Learning One-for-All Multi-Budget Expert Pruning for Flexible MoE Deployment
- OptiMind: Improving Small Language Models for Optimization Modeling through Formulation Guidance and Multi-Turn Learning
- Staying in the Sweet Spot: Boost Reasoning Evolution via Responsive Rollout Difficulty Manipulation
- Examining the Impact of Data and Training Methods on Emergent and Subliminal Misalignment
- Rethinking Black-Box On-Policy Distillation: A Rubric-based Analysis