Skip to content
Last Dictionary
Plane
Mathematics & StatisticsMachine LearningGenerative Media

normal distribution

nounAlso: Gaussian distribution, Gaussian, bell curve, normal curve, multivariate normal distribution

In plain wordsThe familiar bell-shaped curve: most values cluster near the average, and values become rapidly rarer the farther they lie from it. Measurement errors and adult heights roughly follow it.

  1. The continuous probability distributionprobability distribution, n.: A complete description of how likely each possible outcome is — for a fair die, one-sixth for each face. A language model produces one of these over its whole vocabulary at every step. with a bell-shaped density fixed by its mean μ and variancevariance, n.: A number that grows the more widely something's possible values scatter away from its average, and is zero if it never varies. Its square root, the standard deviation, uses the same units as the values themselves. σ²; about 68 percent of its mass lies within one standard deviation of the mean and 99.7 percent within three. The multivariate form is specified by a mean vector and a covariancecovariance, n.: A measure of whether two quantities tend to rise and fall together, move in opposite directions, or show no consistent relationship. matrix. The central limit theoremcentral limit theorem, n.: The mathematical reason bell curves are everywhere: add up many small independent random effects and the total is almost always bell-shaped, whatever the individual effects look like. explains why sums of many small independent effects are approximately normal, and among distributions with a given mean and variance it has maximal entropyentropy, n.: A measure of unpredictability. A fair coin toss has more entropy than a rigged one; an AI that is unsure of its next word has high entropy.. In AI it is everywhere: weights are initialized from normal or truncated-normal distributions; diffusion modelsdiffusion model, n.: An AI that learns to create images, sound or video by practicing noise removal: shown pictures buried under static, it learns to clean them up, then builds new ones starting from pure static. add Gaussian noise step by step and learn to remove it; variational autoencodersvariational autoencoder, n.: A neural network that learns to squeeze data into a compact, slightly fuzzy code and rebuild it, arranged so that decoding random codes produces new, plausible examples. use Gaussian latent variables with the reparameterization trickreparameterization trick, n.: A trick that rewrites a random draw as fixed numbers plus separate, outside randomness, so that a neural network can still learn through a step that involves chance.; Gaussian processesGaussian process, n.: A way of predicting an unknown curve or surface that gives not just a best guess at each point but an honest error bar, narrow near measured points and wide far from them. put it over functions; and squared-error loss is maximum likelihood estimationmaximum likelihood estimation, n.: Tuning a model's settings until the real data it learns from looks as likely, and so as unsurprising, to the model as possible. Most AI training, including the pretraining of chatbots, is a version of this. under Gaussian noise. Real data often have heavier tails, so Gaussian assumptions understate extremes.

    Initialize the embedding table from a normal distribution with standard deviation 0.02.

N(x∣μ,σ2)=12πσ2exp⁡ ⁣(−(x−μ)22σ2)\mathcal{N}(x \mid \mu, \sigma^2) = \frac{1}{\sqrt{2\pi\sigma^2}}\exp\!\Big(-\frac{(x-\mu)^2}{2\sigma^2}\Big)

Origin

Abraham de Moivre derived the curve in 1733 as an approximation to the binomial; Carl Friedrich Gauss (1809) used it for errors of observation, hence 'Gaussian', and Pierre-Simon Laplace (1810) proved an early central limit theorem. Charles Sanders Peirce, Francis Galton and Wilhelm Lexis called it 'normal' in the 1870s, and Karl Pearson's systematic use of 'normal curve' from 1894 helped fix the name.

Usage note

'Normal' here means standard or typical, not 'correct', and is unrelated to a normnorm, n.: A way of measuring the size, or length, of a list of numbers — for example the straight-line distance from the origin to the point the list describes. or to normalization layers.

Related

See also
  • central limit theoremcentral limit theorem, n.: The mathematical reason bell curves are everywhere: add up many small independent random effects and the total is almost always bell-shaped, whatever the individual effects look like.
  • variancevariance, n.: A number that grows the more widely something's possible values scatter away from its average, and is zero if it never varies. Its square root, the standard deviation, uses the same units as the values themselves.
  • probability density functionprobability density function, n.: A curve showing where a measurement is likely to fall. The area under the curve between two values gives the chance of landing between them.
  • diffusion modeldiffusion model, n.: An AI that learns to create images, sound or video by practicing noise removal: shown pictures buried under static, it learns to clean them up, then builds new ones starting from pure static.
  • Gaussian processGaussian process, n.: A way of predicting an unknown curve or surface that gives not just a best guess at each point but an honest error bar, narrow near measured points and wide far from them.
  • maximum likelihood estimationmaximum likelihood estimation, n.: Tuning a model's settings until the real data it learns from looks as likely, and so as unsurprising, to the model as possible. Most AI training, including the pretraining of chatbots, is a version of this.
  • covariancecovariance, n.: A measure of whether two quantities tend to rise and fall together, move in opposite directions, or show no consistent relationship.

Mentioned in

central limit theoremcentral limit theorem, n.: The mathematical reason bell curves are everywhere: add up many small independent random effects and the total is almost always bell-shaped, whatever the individual effects look like., Gaussian mixture modelGaussian mixture model, n.: A model that explains data as coming from several overlapping bell-shaped clouds, working out where each cloud sits, how spread out it is, and how likely each point is to belong to it., GELUGELU, abbr.: A smooth version of the ReLU rule, used in models such as BERT and GPT: it keeps large positive values, removes large negative ones, and blends softly in between., normnorm, n.: A way of measuring the size, or length, of a list of numbers — for example the straight-line distance from the origin to the point the list describes., probability distributionprobability distribution, n.: A complete description of how likely each possible outcome is — for a fair die, one-sixth for each face. A language model produces one of these over its whole vocabulary at every step., variancevariance, n.: A number that grows the more widely something's possible values scatter away from its average, and is zero if it never varies. Its square root, the standard deviation, uses the same units as the values themselves., covariancecovariance, n.: A measure of whether two quantities tend to rise and fall together, move in opposite directions, or show no consistent relationship., probability density functionprobability density function, n.: A curve showing where a measurement is likely to fall. The area under the curve between two values gives the chance of landing between them.