Scalable Frequency- and Length-Aware Subdocument Deduplication for Large Language Model Pretraining

arXiv:2608.03089 · cs.CL · Submitted 2026-08-04 · Read on arXiv

Hai Wang, Chenhao Wang, Qifeng Cai, Yixiu Liu, Miao Peng, Nuo Chen, Yuanlin Tu, Chengcheng Xu, Feng Zhang

cs.CL

Submitted: 2026-08-04

License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/

Terminology

• Benchmark — 19× in this paper · explained in 2 episodes
• Redundancy — 7× in this paper · explained in 2 episodes
• Coherence — 3× in this paper · explained in 2 episodes
• Abstraction — 2× in this paper · explained in 1 episode
• Accuracy — 2× in this paper · explained in 1 episode
• Natural Questions — 2× in this paper
• Papers — 2× in this paper · explained in 3 episodes
• Adam — 1× in this paper · explained in 3 episodes
• Benchmarking — 1× in this paper · explained in 4 episodes
• Composition — 1× in this paper · explained in 2 episodes
• Core — 1× in this paper · explained in 2 episodes
• Frequency Dependence — 1× in this paper · explained in 2 episodes
• Interpretability — 1× in this paper · explained in 15 episodes
• Memorization — 1× in this paper · explained in 4 episodes
• Scaling Laws — 1× in this paper · explained in 4 episodes
• Structural Integrity — 1× in this paper · explained in 3 episodes
• Verifier — 1× in this paper · explained in 2 episodes

Sources

• FullStack Bench: Evaluating LLMs as Full Stack Coders
• Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge
• The Llama 3 Herd of Models
• SoftDedup: an Efficient Data Reweighting Method for Speeding Up Language Model Pre-training
• Measuring Massive Multitask Language Understanding
• Scaling Laws and Interpretability of Learning from Repeated Data
• LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code
• The Stack: 3 TB of permissively licensed source code
• CMMLU: Measuring massive multitask language understanding in Chinese
• DataComp-LM: In search of the next generation of training sets for language models
• Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generation
• Scaling Data-Constrained Language Models
• The RefinedWeb Dataset for Falcon LLM: Outperforming Curated Corpora with Web Data, and Web Data Only
• The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scale
• FineWeb2: One Pipeline to Scale Them All -- Adapting Pre-Training Data Processing to Every Language
• CCNet: Extracting High Quality Monolingual Datasets from Web Crawl Data
• HellaSwag: Can a Machine Really Finish Your Sentence?
• Hy-MT2: A Family of Fast, Efficient and Powerful Multilingual Translation Models in the Wild
• BigCodeBench: Benchmarking Code Generation with Diverse Function Calls and Complex Instructions

Related papers

• Exploring Solution Divergence and Its Effect on Large Language Model Problem Solving
• Ishigaki-IDS-Bench: A Benchmark for Generating Information Delivery Specification from BIM Information Requirements
• Subliminal Steering: Stronger Encoding of Hidden Signals
• MedStruct-S: A Benchmark for Key Discovery, Key-Conditioned QA and Semi-Structured Extraction from OCR Clinical Reports
• The End of Transformers? On Challenging Attention and the Rise of Sub-Quadratic Architectures
• Untangling the Mechanisms of Misleading Context in Medical Question Answering