Selecting Repetition Counts Across Model Scales in Data-Constrained Pretraining
cs.CL, stat.ML
Submitted: 2026-10-04
Updated: 2026-10-04
License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
Sources
- $M^3$ Scaling Law: Optimizing Multi-Epoch, Multi-Lingual, and Multi-Stage Training for Low-Resource Language Models
- Broken Neural Scaling Laws
- DoGE: Domain Reweighting with Generalization Estimation
- BiMix: A Bivariate Data Mixing Law for Language Model Pretraining
- Scaling Laws and Interpretability of Learning from Repeated Data
- Training Compute-Optimal Large Language Models
- Scaling Laws for Neural Language Models
- RegMix: Data Mixture as Regression for Language Model Pre-training
- DataDecide: How to Predict Best Pretraining Data with Small Experiments
- Scaling Data-Constrained Language Models
- Estimating Training Data Influence by Tracing Gradient Descent
- D-CPT Law: Domain-specific Continual Pre-Training Scaling Law for Large Language Models
- Scaling Laws for Mixture Pretraining Under Data Constraints
- Tensor Programs V: Tuning Large Neural Networks via Zero-Shot Hyperparameter Transfer
- Data Mixing Laws: Optimizing Data Mixtures by Predicting Language Modeling Performance
- Data Mixture Optimization: A Multi-fidelity Multi-scale Bayesian Framework
Related papers
- Exploring Solution Divergence and Its Effect on Large Language Model Problem Solving
- Ishigaki-IDS-Bench: A Benchmark for Generating Information Delivery Specification from BIM Information Requirements
- Subliminal Steering: Stronger Encoding of Hidden Signals
- MedStruct-S: A Benchmark for Key Discovery, Key-Conditioned QA and Semi-Structured Extraction from OCR Clinical Reports
- The End of Transformers? On Challenging Attention and the Rise of Sub-Quadratic Architectures
- Untangling the Mechanisms of Misleading Context in Medical Question Answering