On the Convergence of Muon and Beyond
cs.LG
Submitted: 2025-09-19
Updated: 2026-09-30
Code: https://github.com/MoonshotAI/Moonlight
Sources
- Muon is Provably Faster with Momentum Variance Reduction
- LiMuon: Light and Fast Muon Optimizer for Large Models
- ASGO: Adaptive Structured Gradient Optimization
- Old Optimizer, New Norm: An Anthology
- MuonEq: Balancing Before Orthogonalization with Lightweight Equilibration
- Muon Optimizes Under Spectral Norm Constraints
- Symbolic Discovery of Optimization Algorithms
- On the Convergence of A Class of Adam-Type Algorithms for Non-Convex Optimization
- Gradient Descent on Neural Networks Typically Occurs at the Edge of Stability
- Momentum-Based Variance Reduction in Non-Convex SGD
- DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligence
- On MUON optimization: From non-convergence to an error analysis with Polar Express and the Newton-Schulz polynomial from implementations
- Implicit Bias of Spectral Descent and Muon on Multiclass Separable Data
- A Novel Convergence Analysis for Algorithms of the Adam Family
- Convergence of Adam for Non-convex Objectives: Relaxed Hyperparameters and Non-ergodic Case
- Training Compute-Optimal Large Language Models
- SUPER-ADAM: Faster and Universal Framework of Adaptive Gradients
- Better LMO-based Momentum Methods with Second-Order Information
- Adam: A Method for Stochastic Optimization
- Understanding Gradient Orthogonalization for Deep Learning via Non-Euclidean Trust-Region Optimization
Related papers
- Polynomial-Augmented Neural Networks (PANNs) with Weak Orthogonality Constraints for Enhanced Function and PDE Approximation
- AIRL-S: Unifying Reinforcement Learning and Search-Based Test-Time Scaling via Adversarial Inverse Reinforcement Learning
- Transformers as Bayesian In-Context Experimenters: Smoothness-Adaptive Efficient ATE Estimation
- Convergence issues in Relational Concept Analysis based on AOC-posets
- Beliefs Beyond Posteriors: The Predictive Cost of ELBO Binding
- Understanding Diffusion Models via Ratio-Based Function Approximation with SignReLU Networks