Muon learns balanced solutions in matrix factorization without slow saddle-to-saddle dynamics
cs.LG
Submitted: 2026-06-29
Updated: 2026-11-06
Code: https://github.com/dkarkada/muon-mfac
License: http://creativecommons.org/licenses/by/4.0/
The gist: Matrix factorization (i.e., problems of the form P, Q M - P Q F squared) is a minimal learning problem that exhibits both nonlinear parameter dynamics and representation learning.
Terminology
Abstract
Matrix factorization (i.e., problems of the form P, Q | M - P Q| F squared) is a minimal learning problem that exhibits both nonlinear parameter dynamics and representation learning. In this setting, we study how parameter trajectories under the Muon optimizer differ from those of gradient descent. We identify three main dynamical differences: (1) Muon avoids the slow saddle-to-saddle dynamics of GD from small initialization. Muon instead learns all the top singular vectors of M at the same rate, and they saturate in order of increasing (rather than decreasing) singular value. (2) Muon does not diverge even when the learning rate exceeds the critical threshold set by the local loss sharpness. This frees the learning rate from the condition number of the problem, enabling rapid convergence via exponential learning rate annealing. (3) Once the weights are aligned with each other and the target, Muon flow conserves the matrix quantity sqrt P P-sqrt Q Q, while gradient flow is known to conserve the matrix P P - Q Q. Despite having distinct conserved quantities, both optimizers find the so-called balanced solution from vanishing initialization. When training from small random initialization, the weights spontaneously align early in training. We derive the alignment rates in simple settings and show that they predict the empirical alignment rates in general. Finally, we exploit structural properties of Muon to construct a learning rate schedule that achieves near-perfect alignment in only two optimization steps.
Sources
- Implicit Regularization in Deep Matrix Factorization
- Neural Networks as Kernel Learners: The Silent Alignment Effect
- Old Optimizer, New Norm: An Anthology
- Spectral Gradient Descent Mitigates Anisotropy-Driven Misalignment: A Case Study in Phase Retrieval
- When do spectral gradient updates help in deep learning?
- Algorithmic Regularization in Learning Deep Homogeneous Models: Layers are Automatically Balanced
- Implicit Bias of Spectral Descent and Muon on Multiclass Separable Data
- Implicit Regularization of Discrete Gradient Dynamics in Linear Neural Networks
- Implicit Regularization in Matrix Factorization
- LoRA: Low-Rank Adaptation of Large Language Models
- Saddle-to-Saddle Dynamics in Deep Linear Networks: Small Initialization Training, Symmetry, and Sparsity
- Gradient descent aligns the layers of deep linear networks
- Uniform Spectral Growth and Convergence of Muon in LoRA-Style Matrix Factorization
- Closed-Form Training Dynamics Reveal Learned Features and Linear Structure in Word2Vec-like Models
- Muon in Associative Memory Learning: Training Dynamics and Scaling Laws
- Towards Resolving the Implicit Bias of Gradient Descent for Matrix Factorization: Greedy Low-Rank Learning
- Muon is Scalable for LLM Training
- Preconditioning Benefits of Spectral Orthogonalization in Muon
- Abide by the Law and Follow the Flow: Conservation Laws for Gradient Flows
- On Alignment in Deep Linear Neural Networks
Related papers
- A Physics-Regulated Neural Framework for Learning 3D Grain Growth Dynamics
- FlexMoE: Learning One-for-All Multi-Budget Expert Pruning for Flexible MoE Deployment
- OptiMind: Improving Small Language Models for Optimization Modeling through Formulation Guidance and Multi-Turn Learning
- Staying in the Sweet Spot: Boost Reasoning Evolution via Responsive Rollout Difficulty Manipulation
- Examining the Impact of Data and Training Methods on Emergent and Subliminal Misalignment
- Rethinking Black-Box On-Policy Distillation: A Rubric-based Analysis