WSqD: A Horizon-Free Learning Rate Schedule for Large Model Training
Jianhao Ma, Yuxin Chen
cs.LG, math.OC, stat.ML
Submitted: 2026-07-12
License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
Terminology
Sources
- Anytime Training with Schedule-Free Spectral Optimization
- Straight to Zero: Why Linearly Decaying the Learning Rate to Zero Works Best for LLMs
- Scaling Optimal LR Across Token Horizons
- Continual evaluation for lifelong learning: Identifying the stability gap
- ScheduleFree+: Scaling Learning-Rate-Free & Schedule-Free Learning to Large Language Models
- Optimal Linear Decay Learning Rate Schedules and Further Refinements
- The Llama 3 Herd of Models
- Efficient Continual Pre-training by Mitigating the Stability Gap
- Continual Pre-Training of Large Language Models: How to (re)warm your model?
- Scaling Laws and Compute-Optimal Training Beyond Fixed Training Durations
- Training Compute-Optimal Large Language Models
- MiniCPM: Unveiling the Potential of Small Language Models with Scalable Training Strategies
- Simple and Scalable Strategies to Continually Pre-train Large Language Models
- Scaling Laws for Neural Language Models
- Gradient Descent's Last Iterate is Often (slightly) Suboptimal
- A simpler approach to obtaining an O(1/t) convergence rate for the projected stochastic subgradient method
- Revisiting the Last-Iterate Convergence of Stochastic Gradient Methods
- Anytime Pretraining: Horizon-Free Learning-Rate Schedules with Weight Averaging
- Mid-Training of Large Language Models: A Survey
- 2 OLMo 2 Furious
Related papers
- Polynomial-Augmented Neural Networks (PANNs) with Weak Orthogonality Constraints for Enhanced Function and PDE Approximation
- AIRL-S: Unifying Reinforcement Learning and Search-Based Test-Time Scaling via Adversarial Inverse Reinforcement Learning
- Transformers as Bayesian In-Context Experimenters: Smoothness-Adaptive Efficient ATE Estimation
- Convergence issues in Relational Concept Analysis based on AOC-posets
- Beliefs Beyond Posteriors: The Predictive Cost of ELBO Binding
- Understanding Diffusion Models via Ratio-Based Function Approximation with SignReLU Networks