Prediction-Only Distillation in Linear and Logistic Regression
Hien Dang, Pratik Patil, Alessandro Rinaldo
math.ST, cs.LG, stat.ML
Submitted: 2026-07-16
Comments: 73 pages, 21 figures, 12 tables
Code: https://github.com/hhd357/prediction_only_
License: http://creativecommons.org/licenses/by/4.0/
The gist: Self-distillation (SD) is typically studied when the student is retrained on the teacher's original training inputs.
Terminology
Abstract
Self-distillation (SD) is typically studied when the student is retrained on the teacher's original training inputs. In many practical deployments, however, the labeled training data are no longer available, and one has access only to the trained predictor and fresh unlabeled covariates. We study SD in this prediction-only regime through a fresh-X prediction-mixed scheme: a pure-distilled student is trained on fresh covariates pseudo-labeled by the teacher, and the final predictor is an affine combination of the teacher and student predictions. For ridge regression under proportional asymptotics, we derive deterministic equivalents for the optimally mixed prediction risk under general anisotropic covariance and deterministic signal. We show that this risk is strictly smaller than the teacher risk for almost every pair of teacher and student regularization levels, including when the fresh covariates are out-of-distribution and even when their covariance is isotropic. We further show that the optimal mixing weight cannot be identified from unlabeled data alone, but can be consistently estimated in a single post-training step using a small independent labeled calibration set, without additional model fitting. Finally, for binary logistic regression, we show that prediction mixing can outperform both the teacher and the pure-distilled classifier.
Sources
- Variational Information Distillation for Knowledge Transfer
- Do Deep Nets Really Need to be Deep?
- From Collapse to Improvement: Statistical Perspectives on the Evolutionary Dynamics of Iterative Training on Contaminated Sources
- Deep learning: a statistical viewpoint
- Weak-to-Strong Generalization: Eliciting Strong Capabilities With Weak Supervision
- Unlabeled Data Improves Adversarial Robustness
- Knowledge Distillation with the Reused Teacher Classifier
- Retraining with Predicted Hard Labels Provably Increases Model Accuracy
- Model Collapse Demystified: The Case of Regression
- Strong Model Collapse
- Born Again Neural Networks
- Preventing Model Collapse Under Overparametrization: Optimal Mixing Ratios for Interpolation Learning and Ridge Regression
- Is Model Collapse Inevitable? Breaking the Curse of Recursion by Accumulating Real and Synthetic Data
- Surprises in High-Dimensional Ridgeless Least Squares Interpolation
- Recursive Learning Without Collapse: A Weighting-Based Stabilization Framework
- Deep Residual Learning for Image Recognition
- Distilling the Knowledge in a Neural Network
- Self-Boost via Optimal Retraining: An Analysis via Approximate Message Passing
- Rethinking Self-Distillation: Label Averaging and Enhanced Soft Label Refinement with Partial Labels
- Anisotropic local laws for random matrices
Related papers
- Conformal Prediction for Dyadic Regression Under Complex Missingness
- Sharp Asymptotic Theory of Maximum Likelihood Estimation for Gaussian Processes with an RBF Kernel
- Bentkus-type asymptotic e-values
- High-Dimensional Asymptotics of Differentially Private PCA
- A proximal gradient algorithm for composite log-concave sampling
- KL Convergence Guarantees for Score diffusion models under minimal data assumptions