A Stability Analysis of AdamW: Unstable Equilibria and Non-Convergence
cs.LG, math.OC, stat.ML
Submitted: 2026-11-08
Updated: 2026-11-08
Code: https://github.com/gowerrobert/GPT-opt4https:
Terminology
Sources
- The Polar Express: Optimal Matrix Sign Methods and Their Application to the Muon Algorithm
- Power Lines: Scaling Laws for Weight Decay and Batch Size in LLM Pre-training
- Lion Secretly Solves Constrained Optimization: As Lyapunov Predicts
- Adaptive Gradient Methods at the Edge of Stability
- Convergence guarantees for RMSProp and ADAM in non-convex optimization and an empirical comparison to Nesterov acceleration
- A Simple Convergence Proof of Adam and Adagrad
- Adam symmetry theorem: characterization of the convergence of the stochastic Adam optimizer
- Sharp higher order convergence rates for the Adam optimizer
- The Llama 3 Herd of Models
- AlphaDecay: Module-wise Weight Decay for Heavy-Tailed Balancing in LLMs
- On Convergence of Adam for Stochastic Optimization under Relaxed Assumptions
- A Comprehensive Framework for Analyzing the Convergence of Adam: Bridging the Gap with SGD
- Adam: A Method for Stochastic Optimization
- Rotational Equilibrium: How Weight Decay Balances Learning Across Neural Networks
- On the O(sqrt d over K 1/4) Convergence Rate of AdamW Measured by 1 Norm
- Adam Reduces a Unique Form of Sharpness: Theoretical Insights Near the Minimizer Manifold
- DeepSeek-V3 Technical Report
- Decoupled Weight Decay Regularization
- Hamiltonian Descent Methods
- The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scale
Related papers
- A Physics-Regulated Neural Framework for Learning 3D Grain Growth Dynamics
- FlexMoE: Learning One-for-All Multi-Budget Expert Pruning for Flexible MoE Deployment
- OptiMind: Improving Small Language Models for Optimization Modeling through Formulation Guidance and Multi-Turn Learning
- Staying in the Sweet Spot: Boost Reasoning Evolution via Responsive Rollout Difficulty Manipulation
- Examining the Impact of Data and Training Methods on Emergent and Subliminal Misalignment
- Rethinking Black-Box On-Policy Distillation: A Rubric-based Analysis