Self-Distillation is Optimal Among Spectral Shrinkage Estimators
math.ST, cs.LG, stat.ME, stat.ML, stat.TH
Submitted: 2026-05-18
Updated: 2026-11-05
Terminology
Sources
- Distilling the Knowledge in a Neural Network
- Goedel-Prover-V2: Scaling Formal Theorem Proving with Scaffolded Data Synthesis and Self-Correction
- Learning from Noisy Labels with Distillation
- Federated Optimization: Distributed Machine Learning for On-Device Intelligence
- Federated Distillation: A Survey
- Method-of-Moments Inference for GLMs and Doubly Robust Functionals under Proportional Asymptotics
- Characterizing Finite-Dimensional Posterior Marginals in High-Dimensional GLMs via Leave-One-Out
- Optimal Unconstrained Self-Distillation in Ridge Regression: Strict Improvements, Precise Asymptotics, and One-Shot Tuning
- Asymptotic Theory of Iterated Empirical Risk Minimization, with Applications to Active Learning
- Why Self-Training Helps and Hurts: Denoising vs. Signal Forgetting
- Distillation $\approx$ Early Stopping? Harvesting Dark Knowledge Utilizing Anisotropic Information Retrieval For Overparameterized Neural Network
- Towards a theory of model distillation
- Statistical and Algorithmic Insights for Semi-supervised Learning with Self-training
- Theoretical Analysis of Self-Training with Deep Networks on Unlabeled Data
- How does unlabeled data improve generalization in self-training? A one-hidden-layer theoretical analysis
- Optimal ridge penalty for real-world high-dimensional data can be zero or negative due to the implicit ridge regularization
Related papers
- Sigmoid-FTRL: Design-Based Adaptive Neyman Allocation for AIPW Estimators
- Nonparametric Regression on Fixed-Cardinality Subsets:Minimax Risk and Random-Design Effects
- Conformal Prediction for Dyadic Regression Under Complex Missingness
- Optimal Linear Functional Estimation for Proportionally High-Dimensional Linear Models
- Semiparametric Efficient Inference under Non-Informative Complex Survey Designs
- Sharp Asymptotic Theory of Maximum Likelihood Estimation for Gaussian Processes with an RBF Kernel