Optimal higher order convergence rates for the MUON optimizer
math.OC
Submitted: 2026-11-11
Updated: 2026-11-11
Code: https://github.com/KellerJordan/modded-nanogpt
Terminology
Sources
- The Polar Express: Optimal Matrix Sign Methods and Their Application to the Muon Algorithm
- Old Optimizer, New Norm: An Anthology
- Convergence guarantees for Muon: New parameter regimes and generalizations
- Asymptotic stability properties and a priori bounds for Adam and other gradient descent optimization methods
- Sharp higher order convergence rates for the Adam optimizer
- On MUON optimization: From non-convergence to an error analysis with Polar Express and the Newton-Schulz polynomial from implementations
- Practical Efficiency of Muon for Pretraining
- Insights on Muon from Simple Quadratics
- Constant Step Sizes for Muon-Type Spectral Optimizers on Strongly Convex Quadratics
- Mathematical Introduction to Deep Learning: Methods, Implementations, and Theory
- Kimi K2: Open Agentic Intelligence
- Adam: A Method for Stochastic Optimization
- Understanding Gradient Orthogonalization for Deep Learning via Non-Euclidean Trust-Region Optimization
- PolarGrad: A Class of Matrix-Gradient Optimizers from a Unifying Preconditioning Perspective
- A Note on the Convergence of Muon
- Last-Iterate Stationarity of Muon with Finitely Many Newton-Schulz Iterations
- Muon is Scalable for LLM Training
- An overview of gradient descent optimization algorithms
- On the Convergence Analysis of Muon
Related papers
- An Accelerated Proximal Gradient Method for Bures-Wasserstein Space
- FOMTools: Development and AI-Assisted Code Generation for First-Order Methods
- Lions and Muons: Optimization via Stochastic Frank-Wolfe under Heavy-Tailed Noise
- Dynamic Service Recommendation with Congestion-Dependent Joining: Near-Optimal and Constant-Factor Approximation Algorithms
- Integrating Learning and Optimization for Open Transport Capacity Pooling under Disruption
- Zeroth-Order Gradient Estimators under Decision-Dependent Noise