Fast Catch-Up, Late Switching: Optimal Batch Size Scheduling via Functional Scaling Laws
cs.LG, cs.AI, cs.CL, math.OC, stat.ML
Submitted: 2026-02-15
Updated: 2026-02-23
Code: https://github.com/karpathy/nanoGPT
Terminology
Sources
- DeepSeek LLM: Scaling Open-Source Language Models with Longtermism
- DeepSeek-V3 Technical Report
- Optimal Linear Decay Learning Rate Schedules and Further Refinements
- Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour
- The Llama 3 Herd of Models
- Scaling Laws for Autoregressive Generative Modeling
- Deep Learning Scaling is Predictable, Empirically
- Scaling Laws for Hyperparameter Optimization
- Scaling Laws for Neural Language Models
- Scaling Laws for Gradient Descent and Sign Descent for Linear Bigram Models under Zipf's Law
- AdAdaGrad: Adaptive Batch Size Schemes for Adaptive Gradient Methods
- Functional Scaling Laws in Kernel Regression: Loss Dynamics and Learning Rate Schedules
- Muon in Associative Memory Learning: Training Dynamics and Scaling Laws
- Optimal Learning Rate Schedules under Functional Scaling Laws: Power Decay and Warmup-Stable-Decay
- Predictable Scale: Part I, Step Law -- Optimal Hyperparameter Scaling Law in Large Language Model Pretraining
- RoBERTa: A Robustly Optimized BERT Pretraining Approach
- LongCat-Flash Technical Report
- An Empirical Model of Large-Batch Training
- Critical Batch Size Revisited: A Simple Empirical Approach to Large-Batch Language Model Training
- MiniMax-01: Scaling Foundation Models with Lightning Attention
Related papers
- A Physics-Regulated Neural Framework for Learning 3D Grain Growth Dynamics
- FlexMoE: Learning One-for-All Multi-Budget Expert Pruning for Flexible MoE Deployment
- OptiMind: Improving Small Language Models for Optimization Modeling through Formulation Guidance and Multi-Turn Learning
- Staying in the Sweet Spot: Boost Reasoning Evolution via Responsive Rollout Difficulty Manipulation
- Examining the Impact of Data and Training Methods on Emergent and Subliminal Misalignment
- Rethinking Black-Box On-Policy Distillation: A Rubric-based Analysis