greedy decoding
In plain wordsThe simplest way for a language model to write: always pick the single most likely next word. It is fast and predictable but can produce dull or repetitive text.
A decodingdecoding, n.: The method a language model uses to turn its predictions into actual text — for example, always picking the most likely next word, or choosing somewhat at random among the likely ones. strategy that selects at every step the single tokentoken, n.: The small chunk of text an AI language model reads and writes — often a word, part of a word or a punctuation mark. AI usage, limits and prices are counted in tokens. with the highest probability (the arg max of the logitslogit, n.: A raw score that a model gives each possible answer before the scores are turned into probabilities. In statistics, the 'log-odds': a probability rewritten on a scale running from minus to plus infinity.), with no randomness and no look-ahead. It is equivalent to sampling at temperaturetemperature, n.: A setting that controls how adventurous an AI model's word choices are. Low temperature makes it pick the safest, most likely words; high temperature makes it more varied and surprising, but more error-prone. zero and to beam searchbeam search, n.: A search method that keeps several of the best partial answers at each step, rather than only the single best, to find a better overall result without trying every possibility. with a beam width of one. Greedy decoding is fast and nominally deterministic, which suits short factual answers, classification-style outputs, code and evaluations where reproducibility matters. Because it commits to locally best choices, it can miss sequences of higher overall probability, and in open-ended generation it is prone to repetitive loops and generic phrasing. Served models can still give different greedy outputs across runs because of numerical nondeterminismnondeterminism, n.: The fact that an AI model can give different answers to exactly the same question, partly because it deliberately chooses words with some randomness and partly because of tiny computing differences. in batched GPU computation.
For the extraction benchmark we use greedy decoding so that scores are comparable across runs.
Origin
Named by analogy with greedy algorithms in computer science, which make the locally optimal choice at each step; the term has been standard in neural sequence generation since sequence-to-sequence models of the mid-2010s.
Usage note
Many APIs offer greedy decoding only as temperature 0 rather than as a separate mode.
Related
- decodingdecoding, n.: The method a language model uses to turn its predictions into actual text — for example, always picking the most likely next word, or choosing somewhat at random among the likely ones.
- beam searchbeam search, n.: A search method that keeps several of the best partial answers at each step, rather than only the single best, to find a better overall result without trying every possibility.
- temperaturetemperature, n.: A setting that controls how adventurous an AI model's word choices are. Low temperature makes it pick the safest, most likely words; high temperature makes it more varied and surprising, but more error-prone.
- nondeterminismnondeterminism, n.: The fact that an AI model can give different answers to exactly the same question, partly because it deliberately chooses words with some randomness and partly because of tiny computing differences.
- top-k samplingtop-k sampling, n.: A way of making AI-written text varied but sensible: at each step the model picks at random, but only among its k most likely next words, such as the top 40.
- beam searchbeam search, n.: A search method that keeps several of the best partial answers at each step, rather than only the single best, to find a better overall result without trying every possibility.
- top-p samplingtop-p sampling, n.: A way of making AI-written text varied but sensible: the model picks at random among the smallest group of likely next words whose chances add up to a set share, such as 90%.
Mentioned in
temperaturetemperature, n.: A setting that controls how adventurous an AI model's word choices are. Low temperature makes it pick the safest, most likely words; high temperature makes it more varied and surprising, but more error-prone., beam searchbeam search, n.: A search method that keeps several of the best partial answers at each step, rather than only the single best, to find a better overall result without trying every possibility., decode phasedecode phase, n.: The second stage of an AI's reply, in which it writes the answer one word-piece at a time, each step depending on everything written so far., decodingdecoding, n.: The method a language model uses to turn its predictions into actual text — for example, always picking the most likely next word, or choosing somewhat at random among the likely ones., nondeterminismnondeterminism, n.: The fact that an AI model can give different answers to exactly the same question, partly because it deliberately chooses words with some randomness and partly because of tiny computing differences., repetition penaltyrepetition penalty, n.: A setting that discourages an AI model from repeating itself by lowering the chances of words it has already used, so its text does not get stuck in loops., samplingsampling, n.: Picking some items at random from a larger set or from a range of possibilities — surveying 1,000 voters, say, or letting an AI choose its next word by a weighted lottery., self-consistencyself-consistency, n.: Asking an AI the same question many times, letting it reason differently each time, then going with the answer it reaches most often — a vote among its own attempts., sequence-to-sequence modelsequence-to-sequence model, n.: A model that turns one sequence into another, such as a French sentence into English or speech into text, by first reading the whole input and then writing the output step by step., speculative decodingspeculative decoding, n.: A speed-up trick in which a small, fast model guesses the next few words and the big model checks all the guesses at once, keeping the right ones: faster answers, same quality., top-k samplingtop-k sampling, n.: A way of making AI-written text varied but sensible: at each step the model picks at random, but only among its k most likely next words, such as the top 40., Viterbi algorithmViterbi algorithm, n.: A method that finds the likeliest hidden sequence behind some observations — such as each word's grammatical role in a sentence — by keeping only the best path to each point instead of trying every combination., maximum a posteriori estimationmaximum a posteriori estimation, n.: Picking the single most believable setting of a model after combining the evidence with what was assumed beforehand. In practice it often amounts to ordinary training plus a penalty that keeps settings modest.