There Will Be a Scientific Theory of Deep Learning
stat.ML, cs.LG
Submitted: 2026-04-23
Updated: 2026-11-11
Comments: 43 pages, 6 figures
Code: https://github.com/microsoft/mup
License: http://creativecommons.org/licenses/by/4.0/
The gist: In this paper, we make the case that a scientific theory of deep learning is emerging.
Terminology
Abstract
In this paper, we make the case that a scientific theory of deep learning is emerging. By this we mean a theory which characterizes important properties and statistics of the training process, hidden representations, final weights, and performance of neural networks. We pull together major strands of ongoing research in deep learning theory and identify five growing bodies of work that point toward such a theory: (a) solvable idealized settings that provide intuition for learning dynamics in realistic systems; (b) tractable limits that reveal insights into fundamental learning phenomena; (c) simple mathematical laws that capture important macroscopic observables; (d) theories of hyperparameters that disentangle them from the rest of the training process, leaving simpler systems behind; and (e) universal behaviors shared across systems and settings which clarify which phenomena call for explanation. Taken together, these bodies of work share certain broad traits: they are concerned with the dynamics of the training process; they primarily seek to describe coarse aggregate statistics; and they emphasize falsifiable quantitative predictions. We argue that the emerging theory is best thought of as a mechanics of the learning process, and suggest the name learning mechanics. We discuss the relationship between this mechanics perspective and other approaches for building a theory of deep learning, including the statistical and information-theoretic perspectives. In particular, we anticipate a symbiotic relationship between learning mechanics and mechanistic interpretability. We also review and address common arguments that fundamental theory will not be possible or is not important. We conclude with a portrait of important open directions in learning mechanics and advice for beginners. We host further introductory materials, perspectives, and open questions at learningmechanics.pub.
Sources
- Tensor Programs V: Tuning Large Neural Networks via Zero-Shot Hyperparameter Transfer
- Scaling and renormalization in high-dimensional regression
- Statistical physics of deep learning: Optimal learning of a multi-layer perceptron near interpolation
- On the origin of neural scaling laws: from random graphs to natural language
- Implicit Gradient Regularization
- A unified theory of feature learning in RNNs and DNNs
- Learning quadratic neural networks in high dimensions: SGD dynamics and scaling laws
- Deep Linear Network Training Dynamics from Random Initialization: Data, Width, Depth, and Hyperparameter Transfer
- Depthwise Hyperparameter Transfer in Residual Networks: Dynamics and Scaling Limit
- A Dynamical Model of Neural Scaling Laws
- Learning curves theory for hierarchically compositional data with power-law distributed features
- Deriving Neural Scaling Laws from the statistics of natural language
- Training Dynamics Impact Post-Training Quantization Robustness
- ResNets of All Shapes and Sizes: Convergence of Training Dynamics in the Large-scale Limit
- Sudden Drops in the Loss: Syntax Acquisition, Phase Transitions, and Simplicity Bias in MLMs
- The Hidden Width of Deep ResNets: Tight Error Bounds and Phase Diagram
- Gradient Descent on Neural Networks Typically Occurs at the Edge of Stability
- Recurrent Neural Networks Learn to Store and Generate Sequences using Non-Linear Representations
- Asymptotics of feature learning in two-layer networks after one gradient-step
- Sparse Autoencoders Find Highly Interpretable Features in Language Models
Related papers
- Uniform-in-Time Weak Propagation-of-Chaos in Shallow Neural Networks
- Diffusion Models Observe Only Gradients: A Geometric Perspective on Score Matching Errors
- Quantifying Error Propagation and Model Collapse in Diffusion Models
- Behavior of prediction performance metrics with rare events
- Statistical Inference and Learning for Shapley Additive Explanations (SHAP)
- Control Variate Design for Score-Function Gradient Estimators in Importance-Weighted Variational Inference