Sharp feature-learning transitions and Bayes-optimal neural scaling laws in extensive-width networks
stat.ML, cond-mat.dis-nn, cond-mat.stat-mech, cs.IT, cs.LG, math.IT
Submitted: 2026-05-11
Updated: 2026-11-17
Code: https://github.com/Minh-Toan/scaling-laws
Terminology
Sources
- Learning quadratic neural networks in high dimensions: SGD dynamics and scaling laws
- Learning curves theory for hierarchically compositional data with power-law distributed features
- A Noise Sensitivity Exponent Controls Large Statistical-to-Computational Gaps in Single- and Multi-Index Models
- Optimal scaling laws in learning hierarchical multi-index models
- Scaling Laws and Spectra of Shallow Neural Networks in the Feature Learning Regime
- Deep Learning Scaling is Predictable, Empirically
- Distilling the Knowledge in a Neural Network
- Training Compute-Optimal Large Language Models
- Learning Curve Theory
- Scaling Laws for Neural Language Models
- Emergence and scaling laws in SGD learning of shallow neural networks
Related papers
- Uniform-in-Time Weak Propagation-of-Chaos in Shallow Neural Networks
- Diffusion Models Observe Only Gradients: A Geometric Perspective on Score Matching Errors
- Quantifying Error Propagation and Model Collapse in Diffusion Models
- Behavior of prediction performance metrics with rare events
- Statistical Inference and Learning for Shapley Additive Explanations (SHAP)
- Control Variate Design for Score-Function Gradient Estimators in Importance-Weighted Variational Inference