pretraining data
In plain wordsThe enormous body of text, code and other material — mostly from the web — that a large AI model first learns from, before it is tuned to follow instructions or chat.
The large, mostly unlabeled corpus used in pretrainingpretraining, n.: The first and usually the most expensive stage of building a modern AI model, in which it learns general patterns from enormous amounts of raw text, images or code before being specialized for particular uses. to give a foundation modelfoundation model, n.: A very large AI model, built once at great cost from vast amounts of text, images or code gathered mostly from the internet, that many products then reuse as their starting point, from chatbots to coding tools. general knowledge and skills through self-supervised learningself-supervised learning, n.: A way for AI to learn from huge amounts of unlabeled data by setting itself fill-in-the-blank puzzles, such as guessing a hidden word or a missing patch of an image., usually next-token predictionnext-token prediction, n.: The simple task most AI chatbots were trained on: given some text, guess the next small piece of it. Repeating that guess, piece by piece, produces whole answers.. For language modelslarge language model, n.: A computer program trained on enormous amounts of text to predict the words that come next. At vast scale, that simple skill lets it chat, answer questions, translate, summarize and write code. it typically combines filtered web text derived from Common CrawlCommon Crawl, prop. n.: A nonprofit's free archive of the public web, begun in 2008 and now refreshed with snapshots of billions of pages every month or two. Most large language models learned much of what they know from text taken from it. (usually the largest share) with source code, books, scientific papers, encyclopedias, forums, mathematics and multilingual text, plus a growing share of synthetic datasynthetic data, n.: Data created by computer programs or AI models rather than collected from the real world — such as practice math problems written by a model, or simulated street scenes for self-driving cars.; multimodal models add image–text pairs, video and audio. Scale grew from about 300 billion training tokens for GPT-3GPT-3, prop. n.: A huge 2020 OpenAI language model that could perform new tasks simply from instructions or a few examples written into its prompt, without retraining. It made prompting a new way of programming. (2020) to over 15 trillion for Llama 3 (2024) and several tens of trillions for later open models. What matters is not only size but the data mixturedata mixture, n.: The recipe of different kinds of data a model is trained on — how much web text, code, books, math, science and other languages — which strongly shapes what the finished model is good at., deduplicationdeduplication, n.: Removing repeated copies of the same or nearly the same content from a dataset, so a model does not over-learn whatever happens to be copied many times online., quality filteringquality filtering, n.: Automatically sorting through huge piles of collected text and keeping only the parts judged useful for training, such as well-written articles, while discarding spam, gibberish and junk., decontamination, and the order of data, including high-quality material saved for late in training. Most developers disclose little about it; the EU AI Act requires providers of general-purpose models to publish a summary of their training content.
We doubled the share of code in the pretraining data and saw reasoning scores rise.
Origin
Follows the pretrain-then-fine-tune paradigm of the late 2010s — ELMo, GPT and BERT (2018) — in which a model first learns from a large unlabeled corpus and is then adapted to tasks; 'pretraining data' became a standard phrase as the corpora themselves became objects of research and dispute.
Usage note
Distinguish pretraining data from post-training data such as instruction datasetsinstruction dataset, n.: A collection of example requests paired with good answers, used to teach a pretrained language model to follow instructions and act as a helpful assistant. and preference datapreference data, n.: Records of which of two or more AI answers people (or other AIs) liked better, collected so that a model can learn what good answers look like., which are far smaller and mostly curated. Whether fresh human-written text can keep pace with demand is the data walldata wall, n.: The worry that AI companies will run out of new human-written text to train ever-larger models on, because the useful public internet has already been largely used up. question.
Related
- pretrainingpretraining, n.: The first and usually the most expensive stage of building a modern AI model, in which it learns general patterns from enormous amounts of raw text, images or code before being specialized for particular uses.
- corpuscorpus, n.: A large collection of written or spoken language, gathered so that people or computers can study it or learn from it — like a library assembled for a language model.
- data mixturedata mixture, n.: The recipe of different kinds of data a model is trained on — how much web text, code, books, math, science and other languages — which strongly shapes what the finished model is good at.
- Common CrawlCommon Crawl, prop. n.: A nonprofit's free archive of the public web, begun in 2008 and now refreshed with snapshots of billions of pages every month or two. Most large language models learned much of what they know from text taken from it.
- quality filteringquality filtering, n.: Automatically sorting through huge piles of collected text and keeping only the parts judged useful for training, such as well-written articles, while discarding spam, gibberish and junk.
- data walldata wall, n.: The worry that AI companies will run out of new human-written text to train ever-larger models on, because the useful public internet has already been largely used up.
- tokens per parametertokens per parameter, n.: How many pieces of text a language model reads during training for each adjustable number inside it — a quick way to tell whether a model was trained on a little or a lot of data for its size.
- instruction datasetinstruction dataset, n.: A collection of example requests paired with good answers, used to teach a pretrained language model to follow instructions and act as a helpful assistant.
Mentioned in
knowledge cutoffknowledge cutoff, n.: The date after which an AI model knows nothing from its training, because its training data stops there. Without a web search tool, it cannot know about later events., pretrainingpretraining, n.: The first and usually the most expensive stage of building a modern AI model, in which it learns general patterns from enormous amounts of raw text, images or code before being specialized for particular uses., training datatraining data, n.: The examples an AI system learns from. Whatever patterns, skills, gaps and biases are in the training data tend to show up in the finished system., Common CrawlCommon Crawl, prop. n.: A nonprofit's free archive of the public web, begun in 2008 and now refreshed with snapshots of billions of pages every month or two. Most large language models learned much of what they know from text taken from it., corpuscorpus, n.: A large collection of written or spoken language, gathered so that people or computers can study it or learn from it — like a library assembled for a language model., data contaminationdata contamination, n.: When the test questions used to grade an AI system were already in its training data, so a high score may reflect memorized answers rather than real ability., data mixturedata mixture, n.: The recipe of different kinds of data a model is trained on — how much web text, code, books, math, science and other languages — which strongly shapes what the finished model is good at., data pipelinedata pipeline, n.: The chain of automated steps that collects raw data, cleans and transforms it, and delivers it in a usable form — the factory line that turns messy inputs into training material., data qualitydata quality, n.: How well a set of data does its job: whether it is accurate, complete, up to date, consistent and representative — and, for AI training, informative enough to teach useful skills., data walldata wall, n.: The worry that AI companies will run out of new human-written text to train ever-larger models on, because the useful public internet has already been largely used up., instruction datasetinstruction dataset, n.: A collection of example requests paired with good answers, used to teach a pretrained language model to follow instructions and act as a helpful assistant., low-resource languagelow-resource language, n.: A language with little digital text, recorded speech or language tools available — often one spoken by millions — which makes AI systems much weaker in it than in English., web crawlerweb crawler, n.: A program that automatically travels the web, following links from page to page and downloading what it finds — used by search engines, archives and AI companies to gather data., BookCorpusBookCorpus, prop. n.: A collection of about 11,000 free self-published novels gathered from the web in 2015, used to train the first GPT model and BERT despite problems with consent and duplicates found later., Books3Books3, prop. n.: A collection of nearly 200,000 pirated e-books, assembled in 2020 and used to train several AI language models, which became central to lawsuits by authors over AI training., Brown CorpusBrown Corpus, prop. n.: The first large collection of English text stored on computer for research: about a million words from 500 American publications of 1961, assembled at Brown University in the 1960s., C4C4, prop. n.: A large collection of English web text, cleaned up from a 2019 web archive snapshot by Google researchers, that became a standard ingredient for training language models., canary stringcanary string, n.: A unique marker planted in data to trace what an AI model has absorbed: a code in test files so they can be kept out of training, or a fake secret hidden in training data to measure memorization., FineWebFineWeb, prop. n.: A huge, free collection of cleaned-up English web text — around 15 trillion word fragments — released by Hugging Face in 2024 with a detailed public account of how it was built., LLM groomingLLM grooming, n.: Flooding the internet with propaganda written mainly for machines rather than people, hoping AI chatbots will soak it up and repeat it to their users., quality filteringquality filtering, n.: Automatically sorting through huge piles of collected text and keeping only the parts judged useful for training, such as well-written articles, while discarding spam, gibberish and junk., The PileThe Pile, prop. n.: A large free collection of English text released in 2020 by the volunteer group EleutherAI, mixing 22 sources from books and science papers to code, used to train early open language models., The StackThe Stack, prop. n.: A large free collection of computer source code from public repositories, selected mainly for permissive licenses and released from 2022 onward by the BigCode project for training coding models., WebTextWebText, prop. n.: The private collection of about eight million web pages, chosen because Reddit users had linked to them, that OpenAI used in 2019 to train GPT-2.