C4
In plain wordsA large collection of English web text, cleaned up from a 2019 web archive snapshot by Google researchers, that became a standard ingredient for training language models.
The Colossal Clean Crawled Corpus, an English text dataset of about 750 GB derived from the April 2019 Common CrawlCommon Crawl, prop. n.: A nonprofit's free archive of the public web, begun in 2008 and now refreshed with snapshots of billions of pages every month or two. Most large language models learned much of what they know from text taken from it. snapshot and released to train T5T5, prop. n.: A 2019 Google model that treats every language task — translating, summarizing, answering, classifying — as turning one piece of text into another, using one shared design.. Its cleaning rules discarded pages a language detector was less than 99% sure were English, pages containing any word from a list of obscene and offensive terms, pages with curly braces (a sign of code) or placeholder text, and pages of fewer than three sentences; within pages it kept only lines of at least five words ending in terminal punctuation, and across the corpus it removed repeated three-sentence spans. A documented version holds about 365 million documents and 156 billion tokens. C4 trained many models of the early 2020s, a multilingual variant (mC4) covers about 100 languages, and its rules became a template for later quality filteringquality filtering, n.: Automatically sorting through huge piles of collected text and keeping only the parts judged useful for training, such as well-written articles, while discarding spam, gibberish and junk..
Our ablation baseline is a 1B-parameter model trained on 100B tokens of C4.
Origin
Introduced by Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee and colleagues at Google in 'Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer' (arXiv, October 2019; JMLR 2020). Jesse Dodge and colleagues' 2021 audit documented its contents in detail.
Usage note
The 2021 audit found patents, US military websites, machine-translated text and benchmark examples in C4, and showed that its word blocklist disproportionately removed text by and about minority groups, such as LGBTQ+ people and speakers of African American English. The name's four Cs give the '4'.
Related
- Common CrawlCommon Crawl, prop. n.: A nonprofit's free archive of the public web, begun in 2008 and now refreshed with snapshots of billions of pages every month or two. Most large language models learned much of what they know from text taken from it.
- T5T5, prop. n.: A 2019 Google model that treats every language task — translating, summarizing, answering, classifying — as turning one piece of text into another, using one shared design.
- quality filteringquality filtering, n.: Automatically sorting through huge piles of collected text and keeping only the parts judged useful for training, such as well-written articles, while discarding spam, gibberish and junk.
- pretraining datapretraining data, n.: The enormous body of text, code and other material — mostly from the web — that a large AI model first learns from, before it is tuned to follow instructions or chat.
- corpuscorpus, n.: A large collection of written or spoken language, gathered so that people or computers can study it or learn from it — like a library assembled for a language model.
- FineWebFineWeb, prop. n.: A huge, free collection of cleaned-up English web text — around 15 trillion word fragments — released by Hugging Face in 2024 with a detailed public account of how it was built.
Sources
- Raffel, C. et al. (2020). Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer. JMLR 21. arXiv:1910.10683. arxiv.org/abs/1910.10683
- Dodge, J. et al. (2021). Documenting Large Webtext Corpora: A Case Study on the Colossal Clean Crawled Corpus. EMNLP 2021. arXiv:2104.08758. arxiv.org/abs/2104.08758
Mentioned in
Common CrawlCommon Crawl, prop. n.: A nonprofit's free archive of the public web, begun in 2008 and now refreshed with snapshots of billions of pages every month or two. Most large language models learned much of what they know from text taken from it., corpuscorpus, n.: A large collection of written or spoken language, gathered so that people or computers can study it or learn from it — like a library assembled for a language model., robots.txtrobots.txt, n.: A small public file in which a website posts its rules for automated visitors, such as search-engine and AI crawlers, listing areas they are asked to stay out of. Obeying it is voluntary., FineWebFineWeb, prop. n.: A huge, free collection of cleaned-up English web text — around 15 trillion word fragments — released by Hugging Face in 2024 with a detailed public account of how it was built., language identificationlanguage identification, n.: Automatically working out which language a piece of text or speech is in — a basic sorting step when building multilingual AI training data from the web., quality filteringquality filtering, n.: Automatically sorting through huge piles of collected text and keeping only the parts judged useful for training, such as well-written articles, while discarding spam, gibberish and junk., T5T5, prop. n.: A 2019 Google model that treats every language task — translating, summarizing, answering, classifying — as turning one piece of text into another, using one shared design., The PileThe Pile, prop. n.: A large free collection of English text released in 2020 by the volunteer group EleutherAI, mixing 22 sources from books and science papers to code, used to train early open language models.