Pretraining Data Scale Reverses the Muon-AdamW Fine-Tuning Gap
cs.LG
Submitted: 2026-11-02
Updated: 2026-11-04
Code: https://github.com/tatsu-lab/stanf
Terminology
Sources
- High-dimensional Asymptotics of Feature Learning: How One Gradient Step Improves the Representation
- Training Verifiers to Solve Math Word Problems
- To Use or not to Use Muon: How Simplicity Bias in Optimizers Matters
- Practical Efficiency of Muon for Pretraining
- GLM-4.5: Agentic, Reasoning, and Coding (ARC) Foundation Models
- An Empirical Investigation of Catastrophic Forgetting in Gradient-Based Neural Networks
- Sharp Capacity Scaling of Spectral Optimizers in Learning Associative Memory
- Kimi K2: Open Agentic Intelligence
- Adam: A Method for Stochastic Optimization
- Muon in Associative Memory Learning: Training Dynamics and Scaling Laws
- Muon is Scalable for LLM Training
- Optimizer-Model Consistency: Full Finetuning with the Same Optimizer as Pretraining Forgets Less
- Muon: Training and Trade-offs with Latent Attention and MoE
- Optimizers Qualitatively Alter Solutions And We Should Leverage This
- Muon Learns More Robust and Transferable Features than Adam
- Convergence Bound and Critical Batch Size of Muon Optimizer
- On the Convergence Analysis of Muon
- Continual Learning with Deep Generative Replay
- Overtrained Language Models Are Harder to Fine-Tune
- Isotropic Curvature Model for Understanding Deep Learning Optimization: Is Gradient Orthogonalization Optimal?
Related papers
- OptiMind: Improving Small Language Models for Optimization Modeling through Formulation Guidance and Multi-Turn Learning
- Staying in the Sweet Spot: Boost Reasoning Evolution via Responsive Rollout Difficulty Manipulation
- Examining the Impact of Data and Training Methods on Emergent and Subliminal Misalignment
- Rethinking Black-Box On-Policy Distillation: A Rubric-based Analysis
- (Mis)generalization of Helpful-Only Fine-tuning
- Provably Learning Diffusion Models under the Manifold Hypothesis: Collapse and Refine