What Is a Repeated Token Worth? The Scaling Geometry of Multi-Epoch Pretraining

cs.CL

Submitted: 2026-10-04

Updated: 2026-10-04

Code: https://github.com/KellerJordan/Muon

License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/

Sources

• Demystifying Training-Time Augmentation for Data-Constrained Language Model Pretraining
• ERNIE 5.0 Technical Report
• Scaling Laws for Data Filtering -- Data Curation cannot be Compute Agnostic
• Scaling Laws for Transfer
• Scaling Laws and Interpretability of Learning from Repeated Data
• Scaling Laws for Neural Language Models
• Scale Dependent Data Duplication
• Adam: A Method for Stochastic Optimization
• InfoLaw: Information Scaling Laws for Large Language Models with Quality-Weighted Mixture Data and Repetition
• Prescriptive Scaling Laws for Data Constrained Training
• gzip Predicts Data-dependent Scaling Laws
• Bridging Compute- and Data-Optimal Pretraining
• Scaling Laws for Mixture Pretraining Under Data Constraints
• Pretraining Language Models with Subword Regularization: An Empirical Study of BPE Dropout in Low-Resource NLP
• When Data Is Scarce: Scaling Sparse Language Models with Repeated Training
• Data-Constrained Language Model Pretraining: Improved Regularization and Scaling Laws
• Repetition Mismatch: Why Data Mixture Experiments Don't Scale and How to Fix Them
• Train Smarter, Not Longer: Memorization-Guided Data Reuse for Efficient LLM Training

Related papers

• Exploring Solution Divergence and Its Effect on Large Language Model Problem Solving
• Ishigaki-IDS-Bench: A Benchmark for Generating Information Delivery Specification from BIM Information Requirements
• Subliminal Steering: Stronger Encoding of Hidden Signals
• MedStruct-S: A Benchmark for Key Discovery, Key-Conditioned QA and Semi-Structured Extraction from OCR Clinical Reports
• The End of Transformers? On Challenging Attention and the Rise of Sub-Quadratic Architectures
• Untangling the Mechanisms of Misleading Context in Medical Question Answering