Tune the Baseline First: Most of Muon's Scaled Weight Decay Gain Is a Tuning Gap, but Its Small Real Advantage Grows With Model Size
cs.CL, cs.LG
Submitted: 2026-11-02
Updated: 2026-11-02
Code: https://github.com/ethxin0011/weight_decay_ladder
Terminology
Sources
- Scale Weight Decay and Train Better
- On Empirical Comparisons of Optimizers for Deep Learning
- Benchmarking Neural Network Training Algorithms
- Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity
- Training Compute-Optimal Large Language Models
- MiniCPM: Unveiling the Potential of Small Language Models with Scalable Training Strategies
- Scaling Laws for Neural Language Models
- Rotational Equilibrium: How Weight Decay Balances Learning Across Neural Networks
- Muon is Scalable for LLM Training
- Scaling Data-Constrained Language Models
- OpenWebMath: An Open Dataset of High-Quality Mathematical Web Text
- The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scale
- Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer
- How to set AdamW's weight decay as you scale model and dataset size
- Small-scale proxies for large-scale Transformer training instabilities
Related papers
- ALARM: Audio-Language Alignment for Reasoning Models
- ThinkSum: Probabilistic reasoning over sets using large language models
- Coherence boosting: When your pretrained language model is not paying enough attention
- From Facts to Insights: A Persona-Driven Dual Memory Framework and Dataset for Role-Playing Agents
- Improving General Role-Playing Agents via Psychology-Grounded Reasoning and Role-Aware Policy Optimization
- The Art of Asking: Multilingual Prompt Optimization for Synthetic Data