corpus
In plain wordsA large collection of written or spoken language, gathered so that people or computers can study it or learn from it — like a library assembled for a language model.
A body of language data — written texts or transcribed speech — collected on purpose and kept in machine-readable form, either for studying how a language is used or for training and evaluating language technology (plural corpora). Linguists traditionally build corpora as balanced samples, choosing texts by genre and source so that findings generalize — the Brown CorpusBrown Corpus, prop. n.: The first large collection of English text stored on computer for research: about a million words from 500 American publications of 1961, assembled at Brown University in the 1960s. fixed 500 text samples across 15 genres — and often enrich them with annotationannotation, n.: Adding information to raw data by hand or with software — marking objects in photos, tagging names in sentences, transcribing speech or ranking chatbot answers — so machines can learn from it. such as part-of-speech tags or syntactic trees. In current AI usage the word most often means the text used to pretrain a large language modellarge language model, n.: A computer program trained on enormous amounts of text to predict the words that come next. At vast scale, that simple skill lets it chat, answer questions, translate, summarize and write code.: web-scale corpora such as C4C4, prop. n.: A large collection of English web text, cleaned up from a 2019 web archive snapshot by Google researchers, that became a standard ingredient for training language models., The PileThe Pile, prop. n.: A large free collection of English text released in 2020 by the volunteer group EleutherAI, mixing 22 sources from books and science papers to code, used to train early open language models. and FineWebFineWeb, prop. n.: A huge, free collection of cleaned-up English web text — around 15 trillion word fragments — released by Hugging Face in 2024 with a detailed public account of how it was built. hold from about 150 billion to more than 15 trillion tokenstoken, n.: The small chunk of text an AI language model reads and writes — often a word, part of a word or a punctuation mark. AI usage, limits and prices are counted in tokens., largely filtered from Common CrawlCommon Crawl, prop. n.: A nonprofit's free archive of the public web, begun in 2008 and now refreshed with snapshots of billions of pages every month or two. Most large language models learned much of what they know from text taken from it. and other sources, valued for size and diversity rather than balanced sampling.
We held out 1% of the corpus to measure perplexity.
Origin
From Latin corpus, 'body'; English has long used it for a complete body of writings. The machine-readable sense dates from the Brown Corpus (1964), the first million-word electronic corpus of English; 'corpus linguistics' became an established label in the 1980s, and natural language processing adopted the word for any large body of training text.
Usage note
'Corpus' implies text or speech; for images and other modalities 'dataset' is the usual word. Linguists distinguish a representative, documented corpus from an opportunistic collection, a distinction that web-scale AI corpora often blur.
Related
- datasetdataset, n.: An organized collection of examples gathered for a purpose — such as photos with their captions, or questions with answers — used to teach or test an AI system.
- pretraining datapretraining data, n.: The enormous body of text, code and other material — mostly from the web — that a large AI model first learns from, before it is tuned to follow instructions or chat.
- Brown CorpusBrown Corpus, prop. n.: The first large collection of English text stored on computer for research: about a million words from 500 American publications of 1961, assembled at Brown University in the 1960s.
- Common CrawlCommon Crawl, prop. n.: A nonprofit's free archive of the public web, begun in 2008 and now refreshed with snapshots of billions of pages every month or two. Most large language models learned much of what they know from text taken from it.
- tokentoken, n.: The small chunk of text an AI language model reads and writes — often a word, part of a word or a punctuation mark. AI usage, limits and prices are counted in tokens.
- natural language processingnatural language processing, n.: The field that teaches computers to handle human language — to read, interpret, translate, summarize and produce text and speech. It is one of the oldest and largest areas of artificial intelligence.
- annotationannotation, n.: Adding information to raw data by hand or with software — marking objects in photos, tagging names in sentences, transcribing speech or ranking chatbot answers — so machines can learn from it.
- datasetdataset, n.: An organized collection of examples gathered for a purpose — such as photos with their captions, or questions with answers — used to teach or test an AI system.
Mentioned in
datadata, n.: Recorded facts, measurements, text, images or sounds that a computer can store and process. AI systems learn their skills from large amounts of data rather than from rules written by hand., datasetdataset, n.: An organized collection of examples gathered for a purpose — such as photos with their captions, or questions with answers — used to teach or test an AI system., natural language processingnatural language processing, n.: The field that teaches computers to handle human language — to read, interpret, translate, summarize and produce text and speech. It is one of the oldest and largest areas of artificial intelligence., computational linguisticscomputational linguistics, n.: The science of human language studied with computers: using computational models to understand how language works, and building the language technology that grows out of that understanding., machine translationmachine translation, n.: Software that renders writing or speech in one language into another language, as online translation services and phone translation apps do., Penn TreebankPenn Treebank, prop. n.: A collection of English text, best known for its million words of newspaper articles, in which experts marked each word's grammatical role and drew each sentence's structure as a tree; it trained and tested a generation of language software., pretraining datapretraining data, n.: The enormous body of text, code and other material — mostly from the web — that a large AI model first learns from, before it is tuned to follow instructions or chat., BookCorpusBookCorpus, prop. n.: A collection of about 11,000 free self-published novels gathered from the web in 2015, used to train the first GPT model and BERT despite problems with consent and duplicates found later., Brown CorpusBrown Corpus, prop. n.: The first large collection of English text stored on computer for research: about a million words from 500 American publications of 1961, assembled at Brown University in the 1960s., C4C4, prop. n.: A large collection of English web text, cleaned up from a 2019 web archive snapshot by Google researchers, that became a standard ingredient for training language models., part-of-speech taggingpart-of-speech tagging, n.: Labeling each word in a sentence with its grammatical role — noun, verb, adjective and so on — as in 'Time (noun) flies (verb) like (preposition) an arrow'., syntactic parsingsyntactic parsing, n.: Working out the grammatical structure of a sentence — which words group together and which word modifies which — usually drawn as a tree., The PileThe Pile, prop. n.: A large free collection of English text released in 2020 by the volunteer group EleutherAI, mixing 22 sources from books and science papers to code, used to train early open language models., WebTextWebText, prop. n.: The private collection of about eight million web pages, chosen because Reddit users had linked to them, that OpenAI used in 2019 to train GPT-2.