Denoise First, Orthogonalize Later: Understanding Momentum in Muon via Spectral Filtering
cs.LG
Submitted: 2026-06-02
Updated: 2026-11-04
Code: https://github.com/KellerJordan/modded-nanogpt
Terminology
Sources
- Spectral Gradient Descent Mitigates Anisotropy-Driven Misalignment: A Case Study in Phase Retrieval
- When do spectral gradient updates help in deep learning?
- Momentum via Primal Averaging: Theoretical Insights and Learning Rate Schedules for Non-Convex Optimization
- RMNP: Row-Momentum Normalized Preconditioning for Scalable Matrix-Based Optimization
- Gradient Descent Happens in a Tiny Subspace
- Low-rank Orthogonalization for Large-scale Matrix Optimization with Applications to Foundation Model Training
- Sharp Capacity Scaling of Spectral Optimizers in Learning Associative Memory
- Understanding Gradient Orthogonalization for Deep Learning via Non-Euclidean Trust-Region Optimization
- Muon in Associative Memory Learning: Training Dynamics and Scaling Laws
- Muon is Scalable for LLM Training
- Preconditioning Benefits of Spectral Orthogonalization in Muon
- Empirical Analysis of the Hessian of Over-Parametrized Neural Networks
- Benchmarking Optimizers for Large Language Model Pretraining
- Pion: A Spectrum-Preserving Optimizer via Orthogonal Equivalence Transformation
- Isotropic Curvature Model for Understanding Deep Learning Optimization: Is Gradient Orthogonalization Optimal?
- LLaMA: Open and Efficient Foundation Language Models
- Orthogonalising gradients to speed up neural network optimisation
- How Muon's Spectral Design Benefits Generalization: A Study on Imbalanced Data
- Muon Outperforms Adam in Tail-End Associative Memory Learning
- Fantastic Pretraining Optimizers and Where to Find Them
Related papers
- A Physics-Regulated Neural Framework for Learning 3D Grain Growth Dynamics
- FlexMoE: Learning One-for-All Multi-Budget Expert Pruning for Flexible MoE Deployment
- OptiMind: Improving Small Language Models for Optimization Modeling through Formulation Guidance and Multi-Turn Learning
- Staying in the Sweet Spot: Boost Reasoning Evolution via Responsive Rollout Difficulty Manipulation
- Examining the Impact of Data and Training Methods on Emergent and Subliminal Misalignment
- Rethinking Black-Box On-Policy Distillation: A Rubric-based Analysis