A Solvable Model of Adaptive Learning Rate Rescaling: Acceleration, Stability & Scaling
stat.ML, cond-mat.dis-nn, cs.LG
Submitted: 2026-10-05
Updated: 2026-10-05
Sources
- Training Compute-Optimal Large Language Models
- Scaling Laws for Neural Language Models
- Explaining Neural Scaling Laws
- A Solvable Model of Neural Scaling Laws
- A Dynamical Model of Neural Scaling Laws
- Phases of Muon: When Muon Eclipses SignSGD
- Adam: A Method for Stochastic Optimization
- SOAP: Improving and Stabilizing Shampoo using Adam
- Disentangling Adaptive Gradient Methods from Learning Rates
- Gradient Descent on Neural Networks Typically Occurs at the Edge of Stability
- Edge of Stochastic Stability: Revisiting the Edge of Stability for SGD
- An Empirical Model of Large-Batch Training
- Measuring the Effects of Data Parallelism on Neural Network Training
- A Defense of the Quadratic Model
- The Deep Bootstrap Framework: Good Online Learners are Good Offline Generalizers
- Deep Learning Scaling is Predictable, Empirically
- Deriving Neural Scaling Laws from the statistics of natural language
- How Feature Learning Can Improve Neural Scaling Laws
- Decoupled Weight Decay Regularization
- Muon is Scalable for LLM Training
Related papers
- Behavior of prediction performance metrics with rare events
- Optimal Estimation of Generic Dynamics by Path-Dependent Neural Jump ODEs
- A Posterior-Dynamics Framework for Imaging Inverse Problems with Pretrained Diffusion Priors
- Efficient quadratic entropy with distance sketches
- Embedding-Bias in Conditional Independence Testing
- From Log-Odds to Shapley Values: An Explanatory Geometry for the Weighted Naive Bayes Classifier