Brown Corpus
In plain wordsThe first large collection of English text stored on computer for research: about a million words from 500 American publications of 1961, assembled at Brown University in the 1960s.
A corpuscorpus, n.: A large collection of written or spoken language, gathered so that people or computers can study it or learn from it — like a library assembled for a language model. of 1,014,312 words of edited American prose that appeared in print during 1961, cut into 500 samples of roughly 2,000 words each and allocated across 15 genres, from press reportage to learned writing and romance fiction. Its explicit sampling design made it the model of a balanced, representative corpus, and its word-frequency counts became standard references in linguistics and psychology. A later version tagged every word with one of about 80 part-of-speech categories, supplying training and test data for early statistical taggers and setting the format for successors such as the Lancaster–Oslo/Bergen (LOB) corpus of British English.
Early part-of-speech taggers were trained and scored on the hand-checked tags of the Brown Corpus.
Origin
Compiled by W. Nelson Francis and Henry Kučera at Brown University in Providence, Rhode Island, with U.S. Office of Education funding in 1963–64. Kučera and Francis's frequency study, Computational Analysis of Present-Day American English (1967), made it widely known.
Usage note
Tiny by today's standards — the pretraining data of a large language model is millions of times larger — but it established that language can be studied from a documented, machine-readable sample.
Related
- corpuscorpus, n.: A large collection of written or spoken language, gathered so that people or computers can study it or learn from it — like a library assembled for a language model.
- annotationannotation, n.: Adding information to raw data by hand or with software — marking objects in photos, tagging names in sentences, transcribing speech or ranking chatbot answers — so machines can learn from it.
- natural language processingnatural language processing, n.: The field that teaches computers to handle human language — to read, interpret, translate, summarize and produce text and speech. It is one of the oldest and largest areas of artificial intelligence.
- pretraining datapretraining data, n.: The enormous body of text, code and other material — mostly from the web — that a large AI model first learns from, before it is tuned to follow instructions or chat.
- datasetdataset, n.: An organized collection of examples gathered for a purpose — such as photos with their captions, or questions with answers — used to teach or test an AI system.
Sources
- Kučera, H. & Francis, W. N. (1967). Computational Analysis of Present-Day American English. Brown University Press.
Mentioned in
annotationannotation, n.: Adding information to raw data by hand or with software — marking objects in photos, tagging names in sentences, transcribing speech or ranking chatbot answers — so machines can learn from it., corpuscorpus, n.: A large collection of written or spoken language, gathered so that people or computers can study it or learn from it — like a library assembled for a language model., Penn TreebankPenn Treebank, prop. n.: A collection of English text, best known for its million words of newspaper articles, in which experts marked each word's grammatical role and drew each sentence's structure as a tree; it trained and tested a generation of language software., part-of-speech taggingpart-of-speech tagging, n.: Labeling each word in a sentence with its grammatical role — noun, verb, adjective and so on — as in 'Time (noun) flies (verb) like (preposition) an arrow'.