FineWeb
In plain wordsA huge, free collection of cleaned-up English web text — around 15 trillion word fragments — released by Hugging Face in 2024 with a detailed public account of how it was built.
An open English web-text dataset for pretrainingpretraining, n.: The first and usually the most expensive stage of building a modern AI model, in which it learns general patterns from enormous amounts of raw text, images or code before being specialized for particular uses. language models, built by Hugging FaceHugging Face, prop. n.: A company whose website is the main public library for sharing AI models and datasets — often called the GitHub of machine learning — and which makes widely used free AI software. from 96 Common CrawlCommon Crawl, prop. n.: A nonprofit's free archive of the public web, begun in 2008 and now refreshed with snapshots of billions of pages every month or two. Most large language models learned much of what they know from text taken from it. snapshots spanning 2013 to April 2024 and released with about 15 trillion tokens; later versions added 2024–25 crawls, reaching about 18.5 trillion tokens by mid-2025. Its pipeline, built with the datatrove library, filters URLs against blocklists, extracts text with trafilatura, keeps documents that a fastText classifier identifies as English, applies Gopher and C4-style rules plus custom filters, runs MinHashMinHash, n.: A shortcut for spotting near-copies among millions of documents: each document is boiled down to a short fingerprint, and similar documents end up with similar fingerprints. deduplicationdeduplication, n.: Removing repeated copies of the same or nearly the same content from a dataset, so a model does not over-learn whatever happens to be copied many times online. within each crawl, and anonymizes email and IP addresses. Each design choice was justified by ablations on small models. FineWeb-Edu (1.3 trillion tokens) keeps the roughly 8% of documents that a classifier, trained on Llama 3 70B's ratings of educational value, scores highly; FineWeb2 extends the recipe to over 1,000 languages.
Swapping our web mix for FineWeb-Edu raised MMLU by four points at the same token budget.
Origin
Released by Hugging Face in April 2024, with FineWeb-Edu following in early June, and described in 'The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scale' by Guilherme Penedo, Hynek Kydlíček, Loubna Ben Allal and colleagues (NeurIPS 2024 Datasets and Benchmarks Track).
Usage note
Released under the ODC-By 1.0 license, subject also to Common Crawl's terms of use. Its documented ablations made it a reference recipe for web-data processing, though the underlying web content was collected without opt-in consent.
Related
- Common CrawlCommon Crawl, prop. n.: A nonprofit's free archive of the public web, begun in 2008 and now refreshed with snapshots of billions of pages every month or two. Most large language models learned much of what they know from text taken from it.
- quality filteringquality filtering, n.: Automatically sorting through huge piles of collected text and keeping only the parts judged useful for training, such as well-written articles, while discarding spam, gibberish and junk.
- deduplicationdeduplication, n.: Removing repeated copies of the same or nearly the same content from a dataset, so a model does not over-learn whatever happens to be copied many times online.
- MinHashMinHash, n.: A shortcut for spotting near-copies among millions of documents: each document is boiled down to a short fingerprint, and similar documents end up with similar fingerprints.
- pretraining datapretraining data, n.: The enormous body of text, code and other material — mostly from the web — that a large AI model first learns from, before it is tuned to follow instructions or chat.
- C4C4, prop. n.: A large collection of English web text, cleaned up from a 2019 web archive snapshot by Google researchers, that became a standard ingredient for training language models.
Sources
- Penedo, G. et al. (2024). The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scale. NeurIPS 2024 Datasets and Benchmarks Track. arXiv:2406.17557. arxiv.org/abs/2406.17557v2
- Penedo, G. et al. (2025). FineWeb2: One Pipeline to Scale Them All — Adapting Pre-Training Data Processing to Every Language. arXiv:2506.20920. arxiv.org/abs/2506.20920
Mentioned in
Common CrawlCommon Crawl, prop. n.: A nonprofit's free archive of the public web, begun in 2008 and now refreshed with snapshots of billions of pages every month or two. Most large language models learned much of what they know from text taken from it., corpuscorpus, n.: A large collection of written or spoken language, gathered so that people or computers can study it or learn from it — like a library assembled for a language model., C4C4, prop. n.: A large collection of English web text, cleaned up from a 2019 web archive snapshot by Google researchers, that became a standard ingredient for training language models., language identificationlanguage identification, n.: Automatically working out which language a piece of text or speech is in — a basic sorting step when building multilingual AI training data from the web., MinHashMinHash, n.: A shortcut for spotting near-copies among millions of documents: each document is boiled down to a short fingerprint, and similar documents end up with similar fingerprints., quality filteringquality filtering, n.: Automatically sorting through huge piles of collected text and keeping only the parts judged useful for training, such as well-written articles, while discarding spam, gibberish and junk.