Statistical Properties of Training & Generalization
Itay Lavie, Noam Levi, Yonatan Kahn
stat.ML, cs.LG, hep-ph, physics.data-an
Submitted: 2026-06-18
Comments: 32 pages, 3 figures. Part of the VERaiPHY initiative
License: http://creativecommons.org/licenses/by/4.0/
The gist: Deep learning has managed to evade numerous intuitions from classical statistics to achieve unprecedented performance on a number of real-world tasks.
Terminology
Abstract
Deep learning has managed to evade numerous intuitions from classical statistics to achieve unprecedented performance on a number of real-world tasks. In this article, we investigate the key features and surprises of deep learning from a physics-informed perspective, taking care to point out and justify where possible the many choices inherent in constructing a deep learning model. In particular, we review the phenomenon of neural scaling laws and discuss their interplay with the constraints and inductive biases which may be present when applying machine learning to problems in physics.
Sources
- Identifying and attacking the saddle point problem in high-dimensional non-convex optimization
- Deep Learning without Poor Local Minima
- Born-Infeld (BI) for AI: Energy-Conserving Descent (ECD) for Optimization
- Improving Energy Conserving Descent for Machine Learning: Theory and Practice
- Optimizers for Stabilizing Likelihood-free Inference
- High-dimensional dynamics of generalization error in neural networks
- Surprises in High-Dimensional Ridgeless Least Squares Interpolation
- To understand deep learning we need to understand kernel learning
- Deep Learning is Not So Mysterious or Different
- Demystifying Spectral Bias on Real-World Data
- From Kernels to Features: A Multi-Scale Adaptive Theory of Feature Learning
- Applications of Statistical Field Theory in Deep Learning
- Adaptive kernel predictors from feature-learning infinite limits of neural networks
- Scaling Laws for Neural Language Models
- Training Compute-Optimal Large Language Models
- Scaling description of generalization with number of parameters in deep learning
- Explaining Neural Scaling Laws
- A Dynamical Model of Neural Scaling Laws
- Deriving Neural Scaling Laws from the statistics of natural language
- Statistical physics of deep learning: Optimal learning of a multi-layer perceptron near interpolation
Related papers
- Behavior of prediction performance metrics with rare events
- Optimal Estimation of Generic Dynamics by Path-Dependent Neural Jump ODEs
- A Posterior-Dynamics Framework for Imaging Inverse Problems with Pretrained Diffusion Priors
- Scalable Posterior Uncertainty for Flexible Density-Based Clustering
- A Transport Geometry of Benign Overfitting
- Efficient quadratic entropy with distance sketches