top-k sampling
In plain wordsA way of making AI-written text varied but sensible: at each step the model picks at random, but only among its k most likely next words, such as the top 40.
A stochastic decodingdecoding, n.: The method a language model uses to turn its predictions into actual text — for example, always picking the most likely next word, or choosing somewhat at random among the likely ones. method that at each step keeps only the k tokenstoken, n.: The small chunk of text an AI language model reads and writes — often a word, part of a word or a punctuation mark. AI usage, limits and prices are counted in tokens. with the highest probability, sets the rest to zero, renormalizes, and samples from the result. Truncation removes the long tail of individually unlikely tokens whose combined probability would otherwise cause frequent incoherent choices. Its weakness is that k is fixed: when the model is confident, k may admit poor candidates, and when many continuations are plausible, it may exclude good ones; top-p samplingtop-p sampling, n.: A way of making AI-written text varied but sensible: the model picks at random among the smallest group of likely next words whose chances add up to a set share, such as 90%. and min-p samplingmin-p sampling, n.: A sampling method that lets the model consider any next word at least a set fraction as likely as its top choice, so the shortlist shrinks when the model is sure and grows when it is not. adapt the cut-off to the shape of the distribution. Top-k is often combined with temperaturetemperature, n.: A setting that controls how adventurous an AI model's word choices are. Low temperature makes it pick the safest, most likely words; high temperature makes it more varied and surprising, but more error-prone. and other truncations, and k = 1 is equivalent to greedy decodinggreedy decoding, n.: The simplest way for a language model to write: always pick the single most likely next word. It is fast and predictable but can produce dull or repetitive text.. The widely shared GPT-2 samples of 2019 were generated with k = 40.
We sample with top-k 50 and temperature 0.8 for the story generator.
Origin
Popularized by Angela Fan, Mike Lewis and Yann Dauphin's 'Hierarchical Neural Story Generation' (2018), which sampled from the 10 most likely candidates, and by OpenAI's GPT-2 release (2019).
Related
- top-p samplingtop-p sampling, n.: A way of making AI-written text varied but sensible: the model picks at random among the smallest group of likely next words whose chances add up to a set share, such as 90%.
- min-p samplingmin-p sampling, n.: A sampling method that lets the model consider any next word at least a set fraction as likely as its top choice, so the shortlist shrinks when the model is sure and grows when it is not.
- temperaturetemperature, n.: A setting that controls how adventurous an AI model's word choices are. Low temperature makes it pick the safest, most likely words; high temperature makes it more varied and surprising, but more error-prone.
- decodingdecoding, n.: The method a language model uses to turn its predictions into actual text — for example, always picking the most likely next word, or choosing somewhat at random among the likely ones.
- greedy decodinggreedy decoding, n.: The simplest way for a language model to write: always pick the single most likely next word. It is fast and predictable but can produce dull or repetitive text.
- top-p samplingtop-p sampling, n.: A way of making AI-written text varied but sensible: the model picks at random among the smallest group of likely next words whose chances add up to a set share, such as 90%.
Sources
- Fan, A., Lewis, M. & Dauphin, Y. (2018). Hierarchical Neural Story Generation. ACL 2018. arxiv.org/abs/1805.04833
Mentioned in
temperaturetemperature, n.: A setting that controls how adventurous an AI model's word choices are. Low temperature makes it pick the safest, most likely words; high temperature makes it more varied and surprising, but more error-prone., categorical distributioncategorical distribution, n.: The probabilities for a single pick among a fixed set of options, like a weighted die. A language model's guess about the next word is one of these, spread over its whole vocabulary., decodingdecoding, n.: The method a language model uses to turn its predictions into actual text — for example, always picking the most likely next word, or choosing somewhat at random among the likely ones., greedy decodinggreedy decoding, n.: The simplest way for a language model to write: always pick the single most likely next word. It is fast and predictable but can produce dull or repetitive text., samplingsampling, n.: Picking some items at random from a larger set or from a range of possibilities — surveying 1,000 voters, say, or letting an AI choose its next word by a weighted lottery., top-p samplingtop-p sampling, n.: A way of making AI-written text varied but sensible: the model picks at random among the smallest group of likely next words whose chances add up to a set share, such as 90%., min-p samplingmin-p sampling, n.: A sampling method that lets the model consider any next word at least a set fraction as likely as its top choice, so the shortlist shrinks when the model is sure and grows when it is not.