Influence Diagnostics in High-dimensional M-estimation: Precise Asymptotics
stat.ML, cs.LG
Submitted: 2026-07-10
Updated: 2026-09-25
License: http://creativecommons.org/licenses/by/4.0/
The gist: The impact of a given training point on a statistical model is classically measured through its leave-one-out influence, which quantifies the effect of its removal from the training set on the model
Terminology
Abstract
The impact of a given training point on a statistical model is classically measured through its leave-one-out influence, which quantifies the effect of its removal from the training set on the model accuracy. While the statistics of leave-one-out influences are well understood in the low-dimensional, large sample limit n to infinity, d=O(1), they become more intricate in high dimensions, as the influence of a given sample develops non-trivial dependencies on all other training samples. For convex M-estimation under Gaussian design, in the high-dimensional limit n d, we show that the distribution of the influences across the training set converges to a limiting measure which we sharply characterize. Building on these results, we provide evidence that influential samples tend to lie close to the decision boundary, thereby making contact with a standard data selection heuristic in active learning.
Sources
- Improving the Scaling Laws of Synthetic Data with Deliberate Practice
- Influence Functions in Deep Learning Are Fragile
- An Automatic Finite-Sample Robustness Metric: When Can Dropping a Little Data Make a Big Difference?
- Asymptotic Theory of Iterated Empirical Risk Minimization, with Applications to Active Learning
- Why Less is More (Sometimes): A Theory of Data Curation
- Asymptotic behavior of unregularized and ridge-regularized high-dimensional robust regression estimators : rigorous results
- Statistical and Computational Guarantees for Influence Diagnostics
- Active Learning on a Budget: Opposite Strategies Suit High and Low Budgets
- Adam: A Method for Stochastic Optimization
- Towards a statistical theory of data selection under weak supervision
- Testing Most Influential Sets
- Concentration of measure and generalized product of random vectors with an application to Hanson-Wright-like inequalities
- Resolvent convergence under heterogeneous second-moment profiles and quadratic-form control
- High-Dimensional Statistics: Reflections on Progress and Open Problems
Related papers
- Behavior of prediction performance metrics with rare events
- Optimal Estimation of Generic Dynamics by Path-Dependent Neural Jump ODEs
- A Posterior-Dynamics Framework for Imaging Inverse Problems with Pretrained Diffusion Priors
- Scalable Posterior Uncertainty for Flexible Density-Based Clustering
- A Transport Geometry of Benign Overfitting
- Efficient quadratic entropy with distance sketches