Muon with Nesterov Momentum: Heavy-Tailed Noise and (Randomized) Inexact Polar Decomposition
math.OC, cs.LG
Submitted: 2026-05-07
Updated: 2026-11-01
Comments: 48 pages, 6 figures, 4 tables
Code: https://github.com/Xiaoran-Cheng/muon-randomized-svd
License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
Terminology
Sources
- On the Convergence of Muon and Beyond
- Low-rank Orthogonalization for Large-scale Matrix Optimization with Applications to Foundation Model Training
- From Gradient Clipping to Normalization for Heavy Tailed SGD
- 94% on CIFAR-10 in 3.29 Seconds on a Single GPU
- Understanding Gradient Orthogonalization for Deep Learning via Non-Euclidean Trust-Region Optimization
- A Note on the Convergence of Muon
- Muon is Scalable for LLM Training
- Gluon: Making Muon & Scion Great Again! (Bridging Theory and Practice of LMO-based Optimizers for LLMs)
- Convergence Bound and Critical Batch Size of Muon Optimizer
- On the Convergence Analysis of Muon
- Beyond the Ideal: Analyzing the Inexact Muon Update
- LLaMA: Open and Efficient Foundation Language Models
- GaLore: Memory-Efficient LLM Training by Gradient Low-Rank Projection
Related papers
- An Accelerated Proximal Gradient Method for Bures-Wasserstein Space
- FOMTools: Development and AI-Assisted Code Generation for First-Order Methods
- Lions and Muons: Optimization via Stochastic Frank-Wolfe under Heavy-Tailed Noise
- Dynamic Service Recommendation with Congestion-Dependent Joining: Near-Optimal and Constant-Factor Approximation Algorithms
- Integrating Learning and Optimization for Open Transport Capacity Pooling under Disruption
- Zeroth-Order Gradient Estimators under Decision-Dependent Noise