Common Corpus: The Largest Collection of Ethical Data for LLM Pre-Training
Common Corpus is a massive, open dataset of about 2 trillion tokens for LLM pre-training, addressing the need for legally compliant data. It aggregates diverse content from six collections—like government documents and open code—and uses rigorous tools for quality control, including PII removal and toxicity detection. This resource offers high multilingual diversity suitable for training models.