How to Allocate Your Tokens? Scaling Laws with Training Steps and Batch Size
Fabian Schaipp
cs.LG, stat.ML
Submitted: 2026-07-01
License: http://creativecommons.org/licenses/by/4.0/
The gist: We propose a scaling law that takes into account model size and training data while explicitly splitting the latter into training steps and batch size (called three-term law).
Terminology
Abstract
We propose a scaling law that takes into account model size and training data while explicitly splitting the latter into training steps and batch size (called three-term law). Fitting the proposed law on a large set of training runs, we find that it correctly recovers the scaling of the optimal batch size. Moreover, because it makes use of training runs with suboptimal batch size, our proposed law can be robustly fit with a significantly smaller amount of training runs. We further show that the three-term law can be used to derive scaling laws for suboptimal batch sizes, and that it matches previous empirical findings related to the critical batch size.
Sources
- Chinchilla Scaling: A replication attempt
- A Foundation Model for the Earth System
- DeepSeek LLM: Scaling Open-Source Language Models with Longtermism
- Practical Efficiency of Muon for Pretraining
- On the Role of Batch Size in Stochastic Conditional Gradient Methods
- Scaling Laws for Neural Language Models
- Understanding Gradient Orthogonalization for Deep Learning via Non-Euclidean Trust-Region Optimization
- Predictable Scale: Part I, Step Law -- Optimal Hyperparameter Scaling Law in Large Language Model Pretraining
- An Empirical Model of Large-Batch Training
- Deriving Hyperparameter Scaling Laws via Modern Optimization Theory
Related papers
- Polynomial-Augmented Neural Networks (PANNs) with Weak Orthogonality Constraints for Enhanced Function and PDE Approximation
- AIRL-S: Unifying Reinforcement Learning and Search-Based Test-Time Scaling via Adversarial Inverse Reinforcement Learning
- Transformers as Bayesian In-Context Experimenters: Smoothness-Adaptive Efficient ATE Estimation
- Convergence issues in Relational Concept Analysis based on AOC-posets
- Beliefs Beyond Posteriors: The Predictive Cost of ELBO Binding
- Understanding Diffusion Models via Ratio-Based Function Approximation with SignReLU Networks