sampling

Statistics

In statistics, the procedure of choosing which members of a population to observe — which voters to poll, which patients to enroll, which web pages to label — when the results from those few will be generalized to all. Probability sampling — simple random, stratified, cluster or systematic — gives every unit a known chance of selection, which makes estimates unbiased and their uncertainty computable, with variancevariance, n.: A number that grows the more widely something's possible values scatter away from its average, and is zero if it never varies. Its square root, the standard deviation, uses the same units as the values themselves. typically shrinking in proportion to 1/n; convenience and other non-probability samples risk sampling biassampling bias, n.: A distortion that arises when the data collected over-represents some kinds of people or situations and under-represents others, so conclusions drawn from it don't hold for everyone.. In machine learning the training set is itself a sample from the data distribution, mini-batches are samples of the training set and evaluation sets sample possible inputs, so every reported score carries sampling error (see confidence intervalconfidence interval, n.: A range of plausible values for something being estimated, such as 'accuracy 82 percent, give or take 2 points', reflecting the uncertainty that comes from measuring only a sample.).

Probabilistic models and generation

Drawing random values from a probability distributionprobability distribution, n.: A complete description of how likely each possible outcome is — for a fair die, one-sixth for each face. A language model produces one of these over its whole vocabulary at every step., either to estimate expectations (Monte Carlo methodMonte Carlo method, n.: Estimating an answer by running many random trials and averaging the results — like estimating the odds in a card game by dealing thousands of hands instead of calculating., Markov chain Monte CarloMarkov chain Monte Carlo, n.: A family of methods for drawing random examples from a complicated probability distribution by taking a long, carefully designed random walk that eventually visits each region in proportion to its probability., importance samplingimportance sampling, n.: A shortcut for estimating averages: draw samples from a convenient source, then reweight each one to correct for how much more or less likely it would be under the source you actually care about.) or to generate outputs. Generative modelsgenerative model, n.: A model that learns what a kind of data looks like — faces, sentences, songs — well enough to produce new examples that resemble the originals without copying them. produce text, images and audio by sampling: a language modellarge language model, n.: A computer program trained on enormous amounts of text to predict the words that come next. At vast scale, that simple skill lets it chat, answer questions, translate, summarize and write code. samples each tokentoken, n.: The small chunk of text an AI language model reads and writes — often a word, part of a word or a punctuation mark. AI usage, limits and prices are counted in tokens. from its predicted categorical distributioncategorical distribution, n.: The probabilities for a single pick among a fixed set of options, like a weighted die. A language model's guess about the next word is one of these, spread over its whole vocabulary., reshaped by temperaturetemperature, n.: A setting that controls how adventurous an AI model's word choices are. Low temperature makes it pick the safest, most likely words; high temperature makes it more varied and surprising, but more error-prone., top-k samplingtop-k sampling, n.: A way of making AI-written text varied but sensible: at each step the model picks at random, but only among its k most likely next words, such as the top 40., top-p samplingtop-p sampling, n.: A way of making AI-written text varied but sensible: the model picks at random among the smallest group of likely next words whose chances add up to a set share, such as 90%. or min-p samplingmin-p sampling, n.: A sampling method that lets the model consider any next word at least a set fraction as likely as its top choice, so the shortlist shrinks when the model is sure and grows when it is not., while diffusion modelsdiffusion model, n.: An AI that learns to create images, sound or video by practicing noise removal: shown pictures buried under static, it learns to clean them up, then builds new ones starting from pure static. sample by iterative denoising with a diffusion samplerdiffusion sampler, n.: The step-by-step routine a diffusion model follows to turn noise into a finished picture; the choice of sampler and number of steps trades speed against quality.. Sampling contrasts with deterministic decodingdecoding, n.: The method a language model uses to turn its predictions into actual text — for example, always picking the most likely next word, or choosing somewhat at random among the likely ones. such as greedy decodinggreedy decoding, n.: The simplest way for a language model to write: always pick the single most likely next word. It is fast and predictable but can produce dull or repetitive text. and beam searchbeam search, n.: A search method that keeps several of the best partial answers at each step, rather than only the single best, to find a better overall result without trying every possibility.; drawing many samples enables best-of-n samplingbest-of-n sampling, n.: Asking an AI the same question several times and keeping whichever answer a scoring model or checker rates highest — a simple way to trade extra computing for better answers., self-consistencyself-consistency, n.: Asking an AI the same question many times, letting it reason differently each time, then going with the answer it reaches most often — a vote among its own attempts. voting and pass@kpass@k, n.: A score for code-writing AI: the chance that at least one of k attempts at a problem is correct, as checked by actually running tests on the code. evaluation.

With sampling on, the same prompt gives a different answer each run, so set temperature to zero for regression tests.

Signal processing

Measuring a continuous signal at discrete, evenly spaced instants. By the Nyquist–Shannon theorem the sample rate — for example 16 kHz for many speech models, 44.1 kHz for CD-quality music — must exceed twice the highest frequency present to avoid aliasing (see Fourier transformFourier transform, n.: A way of breaking a signal — a sound, an image, any pattern over time or space — into the simple waves of different frequencies that add up to it, much as a prism splits white light into colors.).