GPT-2
In plain wordsA 2019 OpenAI language model that wrote surprisingly fluent paragraphs. OpenAI at first withheld the full version, saying it could be misused for fake news, which made it famous.
A transformertransformer, n.: The design behind today's chatbots and much of modern AI. It handles text by letting every word look at every other word and decide which of them matter for understanding it. language modellanguage model, n.: A system that has learned how likely different sequences of words are, so it can predict the next word in a text and, by doing that repeatedly, write text of its own. released by OpenAIOpenAI, prop. n.: The American company that makes ChatGPT and the GPT models. It began in 2015 as a nonprofit research lab and is now a for-profit company overseen by a nonprofit foundation. in 2019, with up to 1.5 billion parameters, trained to predict the next word on WebText, about 40 gigabytes of text from some 8 million web pages linked from Reddit posts that had received at least three karma. Scaled up roughly tenfold from the original GPTGPT, abbr.: Short for 'generative pre-trained transformer', the family of OpenAI models behind ChatGPT. The letters are also used loosely for any similar chatbot model., it generated multi-paragraph text coherent enough to startle readers, and it attempted reading comprehension, translation, summarization and question answering with no task-specific training, supporting its paper's claim that language models are unsupervised multitask learners. OpenAI at first released only its smallest version, citing risks of automated disinformation and spam, and published larger versions in stages, with the full model in November 2019. This 'staged release' began a lasting debate over openness and disclosure norms in AI.
GPT-2's story about unicorns in the Andes was the first AI-written text many people took seriously.
Origin
Announced on 14 February 2019 in the blog post 'Better Language Models and Their Implications' and the paper 'Language Models are Unsupervised Multitask Learners' by Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario AmodeiDario Amodei, prop. n.: An American AI researcher who co-founded Anthropic and is its chief executive, known for early work on scaling and AI safety and for forecasting very rapid AI progress. and Ilya SutskeverIlya Sutskever, prop. n.: A Russian-born Israeli-Canadian computer scientist who co-built the AlexNet image recognizer, helped found OpenAI and lead its research until 2024, and now heads the startup Safe Superintelligence Inc..
Usage note
Critics called the withholding a publicity stunt; supporters saw it as a useful experiment in setting norms. Its published weights later made it a standard object of study in mechanistic interpretabilitymechanistic interpretability, n.: A branch of AI research that tries to reverse-engineer a neural network — working out, piece by piece, which internal parts represent which concepts and how they combine to produce behavior..
Related
- GPTGPT, abbr.: Short for 'generative pre-trained transformer', the family of OpenAI models behind ChatGPT. The letters are also used loosely for any similar chatbot model.
- GPT-3GPT-3, prop. n.: A huge 2020 OpenAI language model that could perform new tasks simply from instructions or a few examples written into its prompt, without retraining. It made prompting a new way of programming.
- large language modellarge language model, n.: A computer program trained on enormous amounts of text to predict the words that come next. At vast scale, that simple skill lets it chat, answer questions, translate, summarize and write code.
- OpenAIOpenAI, prop. n.: The American company that makes ChatGPT and the GPT models. It began in 2015 as a nonprofit research lab and is now a for-profit company overseen by a nonprofit foundation.
- open-weight modelopen-weight model, n.: An AI model whose trained internal numbers are published for anyone to download and run on their own computers, even if the data and code used to make it stay private.
- language modellanguage model, n.: A system that has learned how likely different sequences of words are, so it can predict the next word in a text and, by doing that repeatedly, write text of its own.
Sources
- Radford, A., Wu, J., Child, R., Luan, D., Amodei, D. & Sutskever, I. (2019). Language Models are Unsupervised Multitask Learners. OpenAI technical report.
Mentioned in
Dario AmodeiDario Amodei, prop. n.: An American AI researcher who co-founded Anthropic and is its chief executive, known for early work on scaling and AI safety and for forecasting very rapid AI progress., GPTGPT, abbr.: Short for 'generative pre-trained transformer', the family of OpenAI models behind ChatGPT. The letters are also used loosely for any similar chatbot model., mechanistic interpretabilitymechanistic interpretability, n.: A branch of AI research that tries to reverse-engineer a neural network — working out, piece by piece, which internal parts represent which concepts and how they combine to produce behavior., OpenAIOpenAI, prop. n.: The American company that makes ChatGPT and the GPT models. It began in 2015 as a nonprofit research lab and is now a for-profit company overseen by a nonprofit foundation., open-weight modelopen-weight model, n.: An AI model whose trained internal numbers are published for anyone to download and run on their own computers, even if the data and code used to make it stay private., promptprompt, n.: The message or instructions you give an AI system — a question, a request, some examples or a document — that it responds to. Clearer prompts generally get better results., Alec RadfordAlec Radford, prop. n.: An American AI researcher who led OpenAI's first GPT language models and its CLIP and Whisper systems, key steps toward today's chatbots and multimodal AI., byte-pair encodingbyte-pair encoding, n.: A popular way to choose the word pieces an AI breaks text into: start from single characters and repeatedly glue together the pair that occurs most often, until the list reaches the desired size., GPT-3GPT-3, prop. n.: A huge 2020 OpenAI language model that could perform new tasks simply from instructions or a few examples written into its prompt, without retraining. It made prompting a new way of programming., gpt-ossgpt-oss, prop. n.: Two free-to-download reasoning models released by OpenAI in August 2025 — its first downloadable language models since 2019 — that anyone may run, modify and use commercially on their own hardware., scaling hypothesisscaling hypothesis, n.: The bet that today's AI methods, simply made bigger with more computing power and data, will keep getting smarter — possibly all the way to human-level intelligence and beyond — without needing fundamentally new ideas., zero-shot learningzero-shot learning, n.: Doing a task without being shown a single example of it — say, an AI sorting reviews into positive and negative just from being told to, or recognizing an animal it never saw in training., indirect object identificationindirect object identification, n.: A fill-in-the-name task — 'When Mary and John went to the store, John gave a drink to …' — whose solution inside GPT-2 became the best-known fully mapped circuit in a language model., plug and play language modelplug and play language model, n.: An early way to steer what a language model writes — its topic or mood — by letting a small add-on classifier nudge the model's internal state at each word, without retraining the model., WebTextWebText, prop. n.: The private collection of about eight million web pages, chosen because Reddit users had linked to them, that OpenAI used in 2019 to train GPT-2.