On the O(sqrt d over K 1/4) Convergence Rate of AdamW Measured by 1 Norm
cs.LG, math.OC
Submitted: 2025-05-17
Updated: 2026-09-27
Comments: V2: NeurIPS Camera-Ready. V3: expand upon the conference version by incorporating the analysis of NAdamW. V4: Relax the coordinate-wise bounded noise variance assumption to the more general bounded noise variance assumption
Code: https://github.com/adonis-dym/Convergence-Rate-AdamW
Project page: http://skylion007.github.io/OpenWebTextCorpus
License: http://creativecommons.org/licenses/by/4.0/
The gist: As the default optimizer for training large language models, AdamW has achieved remarkable success in deep learning.
Terminology
Abstract
As the default optimizer for training large language models, AdamW has achieved remarkable success in deep learning. However, its convergence behavior is not theoretically well-understood. This paper establishes the convergence rate 1 over K sum k=1 KE [grad f(x k) 1] at most O(sqrt d C over K 1/4) for AdamW measured by 1 norm, where K represents the iteration number, d denotes the model dimension, and C matches the constant in the optimal convergence rate of SGD. Theoretically, we have grad f(x) 2 grad f(x) 1 at most sqrt d grad f(x) 2 for any high-dimensional vector x and E [grad f(x) 1] at least sqrt π E [grad f(x) 2] when each element of grad f(x) is generated from Gaussian distribution N(0,1). Empirically, our experimental results on real-world deep learning tasks reveal grad f(x) 1= (sqrt d) grad f(x) 2. Both support that our convergence rate can be considered to be analogous to the optimal 1 over K sum k=1 KE [grad f(x k) 2] at most O(C over K 1/4) convergence rate of SGD in the ideal case. We also extend our result to NAdamW, an AdamW variant that employs a double-momentum mechanism, and demonstrate that it maintains the same convergence rate.
Sources
- A Novel Convergence Analysis for Algorithms of the Adam Family
- High Probability Convergence of Adam Under Unbounded Gradients and Affine Variance Noise
- Provable Complexity Improvement of AdaGrad over SGD: Upper and Lower Bounds in Stochastic Non-Convex Optimization
- On the $O(\frac{\sqrt{d}}{T^{1/4}})$ Convergence Rate of RMSProp and Its Momentum Extension Measured by $\ell_1$ Norm
- AdaGrad under Anisotropic Smoothness
- Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism
- Reevaluating Theoretical Analysis Methods for Optimization in Deep Learning
- On the Convergence of Adam under Non-uniform Smoothness: Separability from SGDM and Beyond
- Fantastic Pretraining Optimizers and Where to Find Them
- Adam Exploits $\ell_\infty$-geometry of Loss Landscape via Coordinate-wise Adaptivity
Related papers
- Polynomial-Augmented Neural Networks (PANNs) with Weak Orthogonality Constraints for Enhanced Function and PDE Approximation
- AIRL-S: Unifying Reinforcement Learning and Search-Based Test-Time Scaling via Adversarial Inverse Reinforcement Learning
- Transformers as Bayesian In-Context Experimenters: Smoothness-Adaptive Efficient ATE Estimation
- Convergence issues in Relational Concept Analysis based on AOC-posets
- Beliefs Beyond Posteriors: The Predictive Cost of ELBO Binding
- Understanding Diffusion Models via Ratio-Based Function Approximation with SignReLU Networks