DisMUSE: An Efficient Distributed Singular-Spectrum-Based Optimization Approach with Theoretical Convergence Guarantee
cs.LG, math.OC
Submitted: 2026-11-06
Updated: 2026-11-06
Code: https://github.com/karpathy/char-rnn
Terminology
Sources
- Dion: Distributed Orthonormalized Updates
- Sparse Communication for Distributed Gradient Descent
- Scalable Second Order Optimization for Deep Learning
- Old Optimizer, New Norm: An Anthology
- On the Convergence of Muon and Beyond
- Muon Optimizes Under Spectral Norm Constraints
- LiMuon: Light and Fast Muon Optimizer for Large Models
- Adam: A Method for Stochastic Optimization
- Understanding Gradient Orthogonalization for Deep Learning via Non-Euclidean Trust-Region Optimization
- PolarGrad: A Class of Matrix-Gradient Optimizers from a Unifying Preconditioning Perspective
- A Note on the Convergence of Muon
- Muon is Scalable for LLM Training
- FedMuon: Accelerating Federated Learning with Matrix Orthogonalization
- COSMOS: A Hybrid Adaptive Optimizer for Memory-Efficient Training of LLMs
- Low-rank Momentum Factorization for Memory Efficient Training
- Training Deep Learning Models with Norm-Constrained LMOs
- Leveraging Coordinate Momentum in SignSGD and Muon: Memory-Optimized Zero-Order
- Adaptive Federated Optimization
- Convergence Bound and Critical Batch Size of Muon Optimizer
- Practical Efficiency of Muon for Pretraining
Related papers
- A Physics-Regulated Neural Framework for Learning 3D Grain Growth Dynamics
- FlexMoE: Learning One-for-All Multi-Budget Expert Pruning for Flexible MoE Deployment
- OptiMind: Improving Small Language Models for Optimization Modeling through Formulation Guidance and Multi-Turn Learning
- Staying in the Sweet Spot: Boost Reasoning Evolution via Responsive Rollout Difficulty Manipulation
- Examining the Impact of Data and Training Methods on Emergent and Subliminal Misalignment
- Rethinking Black-Box On-Policy Distillation: A Rubric-based Analysis