BookCorpus
In plain wordsA collection of about 11,000 free self-published novels gathered from the web in 2015, used to train the first GPT model and BERT despite problems with consent and duplicates found later.
A dataset of roughly 11,000 unpublished books scraped from the self-publishing site Smashwords, where they were offered free, totaling about 74 million sentences and nearly a billion words. Built for a vision-and-language project that aligned books with their film adaptations, it became important for language modeling because it offered long, continuous narrative text: it was the pretraining corpus of OpenAI's first GPTGPT, abbr.: Short for 'generative pre-trained transformer', the family of OpenAI models behind ChatGPT. The letters are also used loosely for any similar chatbot model. model (2018) and, with English Wikipedia, of BERTBERT, prop. n.: An influential 2018 Google language model that learned to understand text by filling in hidden words. It made search engines and many language tools much better at grasping meaning in context. (2018). A 2021 retrospective datasheet found thousands of duplicated books, a strong skew toward romance and certain other genres, and evidence that many books' terms forbade such copying, making BookCorpus a standard example of what its authors called documentation debt.
BERT's 3.3-billion-word pretraining set combined BookCorpus with English Wikipedia.
Origin
Introduced by Yukun Zhu, Ryan Kiros, Richard Zemel, Ruslan Salakhutdinov, Raquel Urtasun, Antonio Torralba and Sanja Fidler in 'Aligning Books and Movies' (ICCV 2015). Jack Bandy and Nicholas Vincent's retrospective datasheet (2021) documented its problems.
Usage note
The original is no longer distributed by its creators; copies circulating online, including the BookCorpus2 component of The PileThe Pile, prop. n.: A large free collection of English text released in 2020 by the volunteer group EleutherAI, mixing 22 sources from books and science papers to code, used to train early open language models., differ from the version used for GPT and BERT.
Related
- GPTGPT, abbr.: Short for 'generative pre-trained transformer', the family of OpenAI models behind ChatGPT. The letters are also used loosely for any similar chatbot model.
- BERTBERT, prop. n.: An influential 2018 Google language model that learned to understand text by filling in hidden words. It made search engines and many language tools much better at grasping meaning in context.
- datasheet for datasetsdatasheet for datasets, n.: A standard document that travels with a dataset, describing why it was made, what is in it, how it was collected and labeled, and what it should and should not be used for.
- pretraining datapretraining data, n.: The enormous body of text, code and other material — mostly from the web — that a large AI model first learns from, before it is tuned to follow instructions or chat.
- Books3Books3, prop. n.: A collection of nearly 200,000 pirated e-books, assembled in 2020 and used to train several AI language models, which became central to lawsuits by authors over AI training.
- corpuscorpus, n.: A large collection of written or spoken language, gathered so that people or computers can study it or learn from it — like a library assembled for a language model.
Sources
- Zhu, Y. et al. (2015). Aligning Books and Movies: Towards Story-like Visual Explanations by Watching Movies and Reading Books. ICCV 2015.
- Bandy, J. & Vincent, N. (2021). Addressing 'Documentation Debt' in Machine Learning Research: A Retrospective Datasheet for BookCorpus. arXiv:2105.05241. arxiv.org/abs/2105.05241
Mentioned in
datasheet for datasetsdatasheet for datasets, n.: A standard document that travels with a dataset, describing why it was made, what is in it, how it was collected and labeled, and what it should and should not be used for.