A Scaling Theory of Multi-Epoch Pretraining: Late-Time Overfitting and Ensembling
cs.LG
Submitted: 2026-11-02
Updated: 2026-11-02
Terminology
Sources
- Disordered Dynamics in High Dimensions: Connections to Random Matrices and Machine Learning
- Rigorous dynamical mean field theory for stochastic gradient descent methods
- Training Compute-Optimal Large Language Models
- Scaling Laws for Neural Language Models
- Pre-training under infinite compute
- Scaling Laws in Linear Regression: Compute, Parameters, and Data
- A Solvable Model of Neural Scaling Laws
- q0: Primitives for Hyper-Epoch Pretraining
- 4+3 Phases of Compute-Optimal Neural Scaling Laws
- The Eigenlearning Framework: A Conservation Law Perspective on Kernel Regression and Wide Neural Networks
- Larger Datasets Can Be Repeated More: A Theoretical Analysis of Multi-Epoch Scaling in Linear Regression
Related papers
- A Physics-Regulated Neural Framework for Learning 3D Grain Growth Dynamics
- FlexMoE: Learning One-for-All Multi-Budget Expert Pruning for Flexible MoE Deployment
- OptiMind: Improving Small Language Models for Optimization Modeling through Formulation Guidance and Multi-Turn Learning
- Staying in the Sweet Spot: Boost Reasoning Evolution via Responsive Rollout Difficulty Manipulation
- Examining the Impact of Data and Training Methods on Emergent and Subliminal Misalignment
- Rethinking Black-Box On-Policy Distillation: A Rubric-based Analysis