LionMuon: Alternating Spectral and Sign Descent for Efficient Training
cs.LG
Submitted: 2026-09-28
Updated: 2026-10-06
Code: https://github.com/brain-lab-research/lion-muon
Sources
- Training Compute-Optimal Large Language Models
- Kimi K2: Open Agentic Intelligence
- Benchmarking Neural Network Training Algorithms
- Preconditioned Norms: A Unified Framework for Steepest Descent, Quasi-Newton and Adaptive Methods
- Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour
- Muon is Scalable for LLM Training
- DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligence
- DMuon: Efficient Distributed Muon Training with Near-Adam Overhead
- Dion: Distributed Orthonormalized Updates
- Leveraging Coordinate Momentum in SignSGD and Muon: Memory-Optimized Zero-Order
- Sign Operator for Coping with Heavy-Tailed Noise in Non-Convex Optimization: High Probability Bounds Under $(L_0, L_1)$-Smoothness
- Sign-Based Optimizers Are Effective Under Heavy-Tailed Noise
- LiMuon: Light and Fast Muon Optimizer for Large Models
- OLion: Approaching the Hadamard Ideal by Intersecting Spectral and $\ell_{\infty}$ Implicit Biases
- The Polar Express: Optimal Matrix Sign Methods and Their Application to the Muon Algorithm
- Accelerating Newton-Schulz Iteration for Orthogonalization via Chebyshev-type Polynomials
- Lions and Muons: Optimization via Stochastic Frank-Wolfe under Heavy-Tailed Noise
- Improving Generalization Performance by Switching from Adam to SGD
- Accelerated Zeroth-order Method for Non-Smooth Stochastic Convex Optimization Problem with Infinite Variance
- A Note on the Convergence of Muon
Related papers
- Polynomial-Augmented Neural Networks (PANNs) with Weak Orthogonality Constraints for Enhanced Function and PDE Approximation
- AIRL-S: Unifying Reinforcement Learning and Search-Based Test-Time Scaling via Adversarial Inverse Reinforcement Learning
- Transformers as Bayesian In-Context Experimenters: Smoothness-Adaptive Efficient ATE Estimation
- Convergence issues in Relational Concept Analysis based on AOC-posets
- Beliefs Beyond Posteriors: The Predictive Cost of ELBO Binding
- Understanding Diffusion Models via Ratio-Based Function Approximation with SignReLU Networks