Scaling Repeated-Data Pretraining: Batch Size, Regularization, and GPU-Time Tradeoffs
cs.LG, cs.CL
Submitted: 2026-10-11
Updated: 2026-10-11
Terminology
Sources
- DeepSeek LLM: Scaling Open-Source Language Models with Longtermism
- Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge
- Cerebras-GPT: Open Compute-Optimal Language Models Trained on the Cerebras Wafer-Scale Cluster
- Scaling Laws for Neural Language Models
- Branch-Train-Merge: Embarrassingly Parallel Training of Expert Language Models
- An Empirical Model of Large-Batch Training
- Scaling Law for Language Models Training Considering Batch Size
Related papers
- Polynomial-Augmented Neural Networks (PANNs) with Weak Orthogonality Constraints for Enhanced Function and PDE Approximation
- AIRL-S: Unifying Reinforcement Learning and Search-Based Test-Time Scaling via Adversarial Inverse Reinforcement Learning
- Transformers as Bayesian In-Context Experimenters: Smoothness-Adaptive Efficient ATE Estimation
- Convergence issues in Relational Concept Analysis based on AOC-posets
- Beliefs Beyond Posteriors: The Predictive Cost of ELBO Binding
- Understanding Diffusion Models via Ratio-Based Function Approximation with SignReLU Networks