When and How SignSGD Outperforms SGD: A Functional Scaling Law Perspective
stat.ML, cs.LG
Submitted: 2026-11-12
Updated: 2026-11-12
Terminology
Sources
- When and How Does Muon Precondition? The Role of Structured Gradient Noise
- DeepSeek LLM: Scaling Open-Source Language Models with Longtermism
- Scaling Law for Stochastic Gradient Descent in Quadratically Parameterized Linear Regression
- MiniCPM: Unveiling the Potential of Small Language Models with Scalable Training Strategies
- Better Convergence Guarantees for Sign-Based Momentum Methods
- Muon in Associative Memory Learning: Training Dynamics and Scaling Laws
- Optimal Learning Rate Schedules under Functional Scaling Laws: Power Decay and Warmup-Stable-Decay
- Muon is Scalable for LLM Training
- A Solvable Model of Neural Scaling Laws
- Phases of Muon: When Muon Eclipses SignSGD
- On the Convergence Analysis of Muon
- Is your batch size the problem? Revisiting the Adam-SGD gap in language modeling
- LLaMA: Open and Efficient Foundation Language Models
- Momentum in large-batch training: Polyak enlarges the critical batch size, Nesterov improves data efficiency
- Adam Exploits $\ell_\infty$-geometry of Loss Landscape via Coordinate-wise Adaptivity
Related papers
- Uniform-in-Time Weak Propagation-of-Chaos in Shallow Neural Networks
- Diffusion Models Observe Only Gradients: A Geometric Perspective on Score Matching Errors
- Quantifying Error Propagation and Model Collapse in Diffusion Models
- Behavior of prediction performance metrics with rare events
- Statistical Inference and Learning for Shapley Additive Explanations (SHAP)
- Control Variate Design for Score-Function Gradient Estimators in Importance-Weighted Variational Inference