quality filtering
In plain wordsAutomatically sorting through huge piles of collected text and keeping only the parts judged useful for training, such as well-written articles, while discarding spam, gibberish and junk.
The removal from a raw corpus of documents judged unlikely to help a model, a central step in building pretraining datapretraining data, n.: The enormous body of text, code and other material — mostly from the web — that a large AI model first learns from, before it is tuned to follow instructions or chat. from web crawls, where most pages are boilerplate, spam, machine-generated or otherwise low in value. Methods fall into three families. Heuristic rules discard documents by length, symbol-to-word ratio, repeated lines, missing punctuation or blocklisted words, as in C4C4, prop. n.: A large collection of English web text, cleaned up from a 2019 web archive snapshot by Google researchers, that became a standard ingredient for training language models. and DeepMind's Gopher rules. Perplexity filtering keeps text that a small reference language model, often trained on Wikipedia, finds probable. Classifier filtering scores documents with a model trained to recognize a target kind of text: GPT-3GPT-3, prop. n.: A huge 2020 OpenAI language model that could perform new tasks simply from instructions or a few examples written into its prompt, without retraining. It made prompting a new way of programming. kept web pages resembling curated reference corpora, and FineWeb-Edu used a classifier trained on a large model's ratings of educational value, keeping about 8% of FineWebFineWeb, prop. n.: A huge, free collection of cleaned-up English web text — around 15 trillion word fragments — released by Hugging Face in 2024 with a detailed public account of how it was built.. Aggressive filtering raises benchmark scores per token but narrows diversity, so thresholds are tuned by training small models on the filtered output.
Raising the classifier threshold from 2 to 3 halved the data but improved MMLU at the 1B scale.
Origin
Rule-based filtering of web text became standard with C4 (2019); CCNet (Guillaume Wenzek and colleagues, 2019) ranked web text by its perplexity under a language model trained on Wikipedia; GPT-3 (2020) popularized classifier filtering; and FineWeb-Edu and DataComp-LM (both 2024) showed that model-based quality classifiers sharply improve results per token.
Usage note
'Quality' means resemblance to whatever reference a filter imitates. A 2022 study of US school newspapers found that a GPT-3-style filter favored writing from wealthier, more educated and more urban areas.
Related
- pretraining datapretraining data, n.: The enormous body of text, code and other material — mostly from the web — that a large AI model first learns from, before it is tuned to follow instructions or chat.
- data qualitydata quality, n.: How well a set of data does its job: whether it is accurate, complete, up to date, consistent and representative — and, for AI training, informative enough to teach useful skills.
- deduplicationdeduplication, n.: Removing repeated copies of the same or nearly the same content from a dataset, so a model does not over-learn whatever happens to be copied many times online.
- data pipelinedata pipeline, n.: The chain of automated steps that collects raw data, cleans and transforms it, and delivers it in a usable form — the factory line that turns messy inputs into training material.
- FineWebFineWeb, prop. n.: A huge, free collection of cleaned-up English web text — around 15 trillion word fragments — released by Hugging Face in 2024 with a detailed public account of how it was built.
- C4C4, prop. n.: A large collection of English web text, cleaned up from a 2019 web archive snapshot by Google researchers, that became a standard ingredient for training language models.
- data mixturedata mixture, n.: The recipe of different kinds of data a model is trained on — how much web text, code, books, math, science and other languages — which strongly shapes what the finished model is good at.
Sources
- Wenzek, G. et al. (2020). CCNet: Extracting High Quality Monolingual Datasets from Web Crawl Data. LREC 2020.
- Gururangan, S. et al. (2022). Whose Language Counts as High Quality? Measuring Language Ideologies in Text Data Selection. EMNLP 2022.
Mentioned in
Common CrawlCommon Crawl, prop. n.: A nonprofit's free archive of the public web, begun in 2008 and now refreshed with snapshots of billions of pages every month or two. Most large language models learned much of what they know from text taken from it., data cleaningdata cleaning, n.: Finding and fixing errors in a dataset — typos, duplicates, missing values, impossible numbers, wrong labels — before using it, because flawed data produces flawed results., data mixturedata mixture, n.: The recipe of different kinds of data a model is trained on — how much web text, code, books, math, science and other languages — which strongly shapes what the finished model is good at., data pipelinedata pipeline, n.: The chain of automated steps that collects raw data, cleans and transforms it, and delivers it in a usable form — the factory line that turns messy inputs into training material., data qualitydata quality, n.: How well a set of data does its job: whether it is accurate, complete, up to date, consistent and representative — and, for AI training, informative enough to teach useful skills., data-centric AIdata-centric AI, n.: An approach to building AI that focuses on improving the data — fixing labels, adding missing cases, cleaning inconsistencies — rather than endlessly tweaking the model., deduplicationdeduplication, n.: Removing repeated copies of the same or nearly the same content from a dataset, so a model does not over-learn whatever happens to be copied many times online., filterfilter, n.: In an image-recognition network, a small grid of learned numbers that acts as a pattern detector, for vertical edges, a patch of fur or a wheel, scanned across the whole picture., pretraining datapretraining data, n.: The enormous body of text, code and other material — mostly from the web — that a large AI model first learns from, before it is tuned to follow instructions or chat., C4C4, prop. n.: A large collection of English web text, cleaned up from a 2019 web archive snapshot by Google researchers, that became a standard ingredient for training language models., data-constrained scalingdata-constrained scaling, n.: How to keep improving AI models when the supply of fresh, high-quality training text runs out — for example by reusing the same data several times, which works well at first and then stops helping., FineWebFineWeb, prop. n.: A huge, free collection of cleaned-up English web text — around 15 trillion word fragments — released by Hugging Face in 2024 with a detailed public account of how it was built., language identificationlanguage identification, n.: Automatically working out which language a piece of text or speech is in — a basic sorting step when building multilingual AI training data from the web., WebTextWebText, prop. n.: The private collection of about eight million web pages, chosen because Reddit users had linked to them, that OpenAI used in 2019 to train GPT-2.