next-token prediction
In plain wordsThe simple task most AI chatbots were trained on: given some text, guess the next small piece of it. Repeating that guess, piece by piece, produces whole answers.
The training objective, and matching generation procedure, in which a model predicts a probability distribution over the next tokentoken, n.: The small chunk of text an AI language model reads and writes — often a word, part of a word or a punctuation mark. AI usage, limits and prices are counted in tokens. given all preceding tokens. In training, a causal language modelcausal language model, n.: A language model that reads text only from left to right, predicting each word from the words before it. The models behind today's chatbots are of this kind. reads text and is penalized by the cross-entropycross-entropy, n.: A score of how surprised a model's predictions are by what actually happens; lower is better. Language models are trained mainly by driving this number down on real text. between its prediction and the token that actually follows, at every position in parallel thanks to the causal maskcausal mask, n.: A rule inside text-generating AI that stops each word from seeing the words after it, so the model learns to predict what comes next without being able to cheat.; in generation, a token is chosen from the prediction (decodingdecoding, n.: The method a language model uses to turn its predictions into actual text — for example, always picking the most likely next word, or choosing somewhat at random among the likely ones.), appended to the input, and the step repeats. The objective is self-supervised — text supplies its own labels — so it scales to trillions of tokens of unlabeled data. Doing it well requires modeling grammar, facts, reasoning patterns and writers' intentions, which is why pretrainingpretraining, n.: The first and usually the most expensive stage of building a modern AI model, in which it learns general patterns from enormous amounts of raw text, images or code before being specialized for particular uses. on it yields broadly capable models. Alternatives studied include multi-token predictionmulti-token prediction, n.: Training a language model to guess several upcoming words at once rather than only the next one, which can help it learn better and later write faster. and diffusiondiffusion language model, n.: A language model that writes by starting from a fully blanked-out answer and filling it in over several rounds, settling many words at once instead of writing strictly left to right. objectives.
Pretraining is just next-token prediction on fifteen trillion tokens; the assistant behavior comes from post-training.
Origin
Predicting the next symbol of text goes back to Claude Shannon's 1951 experiments on the entropy of English. As a neural-network training objective it appears in Jeffrey Elman's simple recurrent networks (1990) and in the neural language model of Yoshua Bengio and colleagues (2000–2003); OpenAI's GPT series (2018–2020) showed that scaling it with transformers produced general-purpose models.
Usage note
Its sufficiency is debated: Yann LeCun has argued that autoregressive prediction compounds errors and cannot plan, while Ilya Sutskever and others argue that predicting text well requires modeling the reality behind it. See fancy autocompletefancy autocomplete, n.: A dismissive description of AI chatbots as nothing more than a souped-up version of the feature on a phone keyboard that suggests your next word. and stochastic parrotstochastic parrot, n.: A critical label for AI language systems, likening them to parrots that echo patterns in the text they were trained on, fluently but with no grasp of what the words mean..
Related
- language modellanguage model, n.: A system that has learned how likely different sequences of words are, so it can predict the next word in a text and, by doing that repeatedly, write text of its own.
- causal language modelcausal language model, n.: A language model that reads text only from left to right, predicting each word from the words before it. The models behind today's chatbots are of this kind.
- autoregressive modelautoregressive model, n.: A model that produces a sequence one piece at a time, each new piece depending on the pieces already produced — like writing a sentence word by word without going back.
- multi-token predictionmulti-token prediction, n.: Training a language model to guess several upcoming words at once rather than only the next one, which can help it learn better and later write faster.
- pretrainingpretraining, n.: The first and usually the most expensive stage of building a modern AI model, in which it learns general patterns from enormous amounts of raw text, images or code before being specialized for particular uses.
- cross-entropycross-entropy, n.: A score of how surprised a model's predictions are by what actually happens; lower is better. Language models are trained mainly by driving this number down on real text.
- fancy autocompletefancy autocomplete, n.: A dismissive description of AI chatbots as nothing more than a souped-up version of the feature on a phone keyboard that suggests your next word.
- masked language modelmasked language model, n.: A model trained by hiding some words in a sentence and learning to guess them from the words on both sides, like a fill-in-the-blanks exercise. BERT is the famous example.
Mentioned in
GPTGPT, abbr.: Short for 'generative pre-trained transformer', the family of OpenAI models behind ChatGPT. The letters are also used loosely for any similar chatbot model., language modellanguage model, n.: A system that has learned how likely different sequences of words are, so it can predict the next word in a text and, by doing that repeatedly, write text of its own., large language modellarge language model, n.: A computer program trained on enormous amounts of text to predict the words that come next. At vast scale, that simple skill lets it chat, answer questions, translate, summarize and write code., loss functionloss function, n.: A score that measures how wrong a model's predictions are. Training means adjusting the model, step by step, to push this number down., pretrainingpretraining, n.: The first and usually the most expensive stage of building a modern AI model, in which it learns general patterns from enormous amounts of raw text, images or code before being specialized for particular uses., audio language modelaudio language model, n.: An AI language model that can take in sound, such as speech, music and everyday noises, and understand or generate it, much as text chatbots handle words., autoregressive modelautoregressive model, n.: A model that produces a sequence one piece at a time, each new piece depending on the pieces already produced — like writing a sentence word by word without going back., base modelbase model, n.: A language model fresh from its first broad training on text, before being taught to follow instructions or chat. It continues whatever text it is given instead of answering., causal language modelcausal language model, n.: A language model that reads text only from left to right, predicting each word from the words before it. The models behind today's chatbots are of this kind., causal maskcausal mask, n.: A rule inside text-generating AI that stops each word from seeing the words after it, so the model learns to predict what comes next without being able to cheat., completioncompletion, n.: The text an AI language model writes in response to a prompt — literally, its continuation of the text it was given., continued pretrainingcontinued pretraining, n.: Taking an already-trained AI model and resuming its broad, read-everything training on new material — a specialty such as medicine or law, a new language, or fresher data — rather than starting from scratch., cross-entropycross-entropy, n.: A score of how surprised a model's predictions are by what actually happens; lower is better. Language models are trained mainly by driving this number down on real text., decoder-only transformerdecoder-only transformer, n.: The kind of transformer behind most chatbots: a single stack that reads text left to right and continues it, without a separate part that first encodes an input., fancy autocompletefancy autocomplete, n.: A dismissive description of AI chatbots as nothing more than a souped-up version of the feature on a phone keyboard that suggests your next word., mid-trainingmid-training, n.: A middle stage in building a language model, between its broad initial training and its tuning into an assistant: it keeps learning the same way, but from a smaller, carefully chosen diet of math, code, reasoning or long documents., multiclass classificationmulticlass classification, n.: Sorting each item into exactly one of three or more categories — for example, deciding which of the ten digits a handwritten number shows., n-gram modeln-gram model, n.: An early type of language model that predicts the next word from just the few words before it, using counts of how often each word followed those words in a large collection of text., objective functionobjective function, n.: The score that an optimization process tries to push as low, or as high, as possible. For an AI, it defines what 'doing well' means during training., perplexityperplexity, n.: A score for how surprised a language model is by real text; lower is better. A perplexity of 10 means guessing each next word is, on average, as hard for the model as guessing a ten-sided die's roll., predictionprediction, n.: A model's output for a given input: its best guess at an unknown answer. In AI it need not be about the future; labeling a photo 'cat' counts as a prediction., predictive processingpredictive processing, n.: The idea that the brain is a prediction machine: it constantly guesses what its senses will report next and learns mainly from its mistakes, so what we perceive is largely its best guess., pretraining datapretraining data, n.: The enormous body of text, code and other material — mostly from the web — that a large AI model first learns from, before it is tuned to follow instructions or chat., RT-2RT-2, prop. n.: A 2023 Google DeepMind model that controlled robot arms with a chatbot-style vision-and-language model, letting robots follow instructions about objects and ideas they had never practiced with.