top-p sampling
In plain wordsA way of making AI-written text varied but sensible: the model picks at random among the smallest group of likely next words whose chances add up to a set share, such as 90%.
A stochastic decodingdecoding, n.: The method a language model uses to turn its predictions into actual text — for example, always picking the most likely next word, or choosing somewhat at random among the likely ones. method that at each step samples from the smallest set of tokenstoken, n.: The small chunk of text an AI language model reads and writes — often a word, part of a word or a punctuation mark. AI usage, limits and prices are counted in tokens. whose cumulative probability reaches a threshold p — the 'nucleus' — after sorting tokens by probability and renormalizing within the set. Unlike top-k samplingtop-k sampling, n.: A way of making AI-written text varied but sensible: at each step the model picks at random, but only among its k most likely next words, such as the top 40., the number of candidates adapts to the model's confidence: a peaked distribution may leave one or two tokens, a flat one hundreds. Typical values of p are 0.9–0.95, often combined with temperaturetemperature, n.: A setting that controls how adventurous an AI model's word choices are. Low temperature makes it pick the safest, most likely words; high temperature makes it more varied and surprising, but more error-prone.. The method was proposed to address 'neural text degeneration': always choosing high-likelihood tokens produces dull, looping text, while sampling from the full distribution lets rare, low-quality tokens slip in. At high temperatures the nucleus can still contain many poor tokens, which motivated min-p samplingmin-p sampling, n.: A sampling method that lets the model consider any next word at least a set fraction as likely as its top choice, so the shortlist shrinks when the model is sure and grows when it is not..
Keep top-p at 0.95 and adjust only the temperature.
Origin
Introduced as 'nucleus sampling' by Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes and Yejin Choi in 'The Curious Case of Neural Text Degeneration' (arXiv, April 2019; ICLR 2020).
Usage note
'Nucleus sampling' and 'top-p sampling' name the same method. Providers often advise changing either temperature or top-p, not both.
Related
- top-k samplingtop-k sampling, n.: A way of making AI-written text varied but sensible: at each step the model picks at random, but only among its k most likely next words, such as the top 40.
- min-p samplingmin-p sampling, n.: A sampling method that lets the model consider any next word at least a set fraction as likely as its top choice, so the shortlist shrinks when the model is sure and grows when it is not.
- temperaturetemperature, n.: A setting that controls how adventurous an AI model's word choices are. Low temperature makes it pick the safest, most likely words; high temperature makes it more varied and surprising, but more error-prone.
- decodingdecoding, n.: The method a language model uses to turn its predictions into actual text — for example, always picking the most likely next word, or choosing somewhat at random among the likely ones.
- repetition penaltyrepetition penalty, n.: A setting that discourages an AI model from repeating itself by lowering the chances of words it has already used, so its text does not get stuck in loops.
- top-k samplingtop-k sampling, n.: A way of making AI-written text varied but sensible: at each step the model picks at random, but only among its k most likely next words, such as the top 40.
Sources
- Holtzman, A., Buys, J., Du, L., Forbes, M. & Choi, Y. (2020). The Curious Case of Neural Text Degeneration. ICLR 2020. arxiv.org/abs/1904.09751
Mentioned in
temperaturetemperature, n.: A setting that controls how adventurous an AI model's word choices are. Low temperature makes it pick the safest, most likely words; high temperature makes it more varied and surprising, but more error-prone., beam searchbeam search, n.: A search method that keeps several of the best partial answers at each step, rather than only the single best, to find a better overall result without trying every possibility., categorical distributioncategorical distribution, n.: The probabilities for a single pick among a fixed set of options, like a weighted die. A language model's guess about the next word is one of these, spread over its whole vocabulary., decodingdecoding, n.: The method a language model uses to turn its predictions into actual text — for example, always picking the most likely next word, or choosing somewhat at random among the likely ones., greedy decodinggreedy decoding, n.: The simplest way for a language model to write: always pick the single most likely next word. It is fast and predictable but can produce dull or repetitive text., parameterparameter, n.: One of the adjustable numbers inside an AI model that are tuned during training. Large language models have billions of them, and together they store what the model has learned., p-valuep-value, n.: A number saying how surprising your data would be if nothing interesting were going on. A small p-value hints at a real effect, but it is not the probability that the effect is real., repetition penaltyrepetition penalty, n.: A setting that discourages an AI model from repeating itself by lowering the chances of words it has already used, so its text does not get stuck in loops., samplingsampling, n.: Picking some items at random from a larger set or from a range of possibilities — surveying 1,000 voters, say, or letting an AI choose its next word by a weighted lottery., top-k samplingtop-k sampling, n.: A way of making AI-written text varied but sensible: at each step the model picks at random, but only among its k most likely next words, such as the top 40., min-p samplingmin-p sampling, n.: A sampling method that lets the model consider any next word at least a set fraction as likely as its top choice, so the shortlist shrinks when the model is sure and grows when it is not.