The Best Optimizer Depends on Batch Size
cs.LG, stat.ML
Submitted: 2026-10-06
Updated: 2026-10-12
Code: https://github.com/KellerJordan/modded-nanogpt
Project page: https://leloykun.github.io/ponder/steepest-descent-crit-bz
License: http://creativecommons.org/licenses/by/4.0/
The gist: A plethora of new adaptive optimizers are designed to efficiently estimate and use minibatch gradient statistics to shape parameter updates, but they are typically benchmarked at a single batch size.
Abstract
A plethora of new adaptive optimizers are designed to efficiently estimate and use minibatch gradient statistics to shape parameter updates, but they are typically benchmarked at a single batch size. Hyperparameter scaling rules promise to preserve performance as batch size and gradient noise change, suggesting that the best optimizer at one batch size should remain the best at another. We challenge this approach to developing and evaluating optimizers by showing: (1) no principled scaling rule for Muon works consistently across training settings, and (2) the best optimizer for language model pretraining changes with batch size even after extensive hyperparameter tuning.
Sources
- Scaling Optimal LR Across Token Horizons
- On Empirical Comparisons of Optimizers for Deep Learning
- Benchmarking Neural Network Training Algorithms
- Practical Efficiency of Muon for Pretraining
- Scaling Exponents Across Parameterizations and Optimizers
- On the Role of Batch Size in Stochastic Conditional Gradient Methods
- Three Factors Influencing Minima in SGD
- SOAP, Muon, and Beyond: Pushing LLM Pretraining Scales
- Adam: A Method for Stochastic Optimization
- Understanding Gradient Orthogonalization for Deep Learning via Non-Euclidean Trust-Region Optimization
- Stochastic Modified Equations and Dynamics of Stochastic Gradient Algorithms I: Mathematical Foundations
- Preconditioned Stochastic Gradient Descent
- Towards Understanding Momentum Acceleration in River-Valley Loss Landscape
- Small Batch Size Training for Language Models: When Vanilla SGD Works, and Why Gradient Accumulation Is Wasteful
- An Empirical Model of Large-Batch Training
- A Defense of the Quadratic Model
- Adaptive Methods through the Lens of SDEs: Theoretical Insights on the Role of Noise
- The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scale
- Hyperparameter Transfer Enables Consistent Gains of Matrix-Preconditioned Optimizers Across Scales
- Convergence Bound and Critical Batch Size of Muon Optimizer
Related papers
- Polynomial-Augmented Neural Networks (PANNs) with Weak Orthogonality Constraints for Enhanced Function and PDE Approximation
- AIRL-S: Unifying Reinforcement Learning and Search-Based Test-Time Scaling via Adversarial Inverse Reinforcement Learning
- Transformers as Bayesian In-Context Experimenters: Smoothness-Adaptive Efficient ATE Estimation
- Convergence issues in Relational Concept Analysis based on AOC-posets
- Beliefs Beyond Posteriors: The Predictive Cost of ELBO Binding
- Understanding Diffusion Models via Ratio-Based Function Approximation with SignReLU Networks