Skip to content
Last Dictionary
Plane
Machine LearningMathematics & Statistics

Gaussian process

nounAlso: GP, Gaussian process regression, GPR, kriging, covariance function

In plain wordsA way of predicting an unknown curve or surface that gives not just a best guess at each point but an honest error bar, narrow near measured points and wide far from them.

  1. A probability distributionprobability distribution, n.: A complete description of how likely each possible outcome is — for a fair die, one-sixth for each face. A language model produces one of these over its whole vocabulary at every step. over functions, fully specified by a mean function and a covariance (kernel) function that encodes assumptions such as smoothness, length scale or periodicity; technically, a random function whose values at any finite set of inputs are jointly Gaussian. In Gaussian process regression, a GP priorprior, n.: What you believe, and how strongly, before looking at the evidence in front of you. In AI it also means built-in assumptions that steer what a system learns. over functions combined with Gaussian observation noise gives a closed-form posteriorposterior, n.: What you believe after weighing new evidence against what you believed before. It is the updated belief that Bayes' theorem produces.: a predictive mean and variance at every input, with uncertainty growing away from the data. Kernel hyperparameters are fit by maximizing the marginal likelihood. GPs are nonparametricnonparametric model, n.: A model that lets the data decide how complex it becomes: the more examples it gets, the more detail it can capture — for instance, one that predicts by comparing a new case with every stored example. and data-efficient but cost O(n³) time in the number of training points, motivating sparse and inducing-point approximations; classification needs approximate inference. They are the standard surrogate model in Bayesian optimizationBayesian optimization, n.: A smart way to tune settings when each test is expensive: build a statistical guess of how settings affect results, then try whichever setting that guess says is most promising next., a staple of geostatistics, and the limit of infinitely wide neural networks at initialization.

    With only 40 expensive simulations, a Gaussian process gave us both a response surface and where to sample next.

f∼GP(m(x), k(x,x′)),fˉ∗=m(x∗)+k∗⊤(K+σn2I)−1(y−m)f \sim \mathcal{GP}\big(m(\mathbf{x}),\, k(\mathbf{x}, \mathbf{x}')\big), \qquad \bar{f}_* = m(\mathbf{x}_*) + \mathbf{k}_*^{\top}(K + \sigma_n^2 I)^{-1}(\mathbf{y} - \mathbf{m})

Origin

Studied in probability theory by the 1940s — Joseph Doob's paper 'The Elementary Gaussian Processes' (1944) uses the name — notably in Andrey Kolmogorov's and Norbert Wiener's work on prediction of stationary processes. Prediction with them emerged in geostatistics as kriging, from Danie Krige's 1951 mining thesis and Georges Matheron's theory (from 1960). Radford Neal (1996) linked them to infinitely wide neural networks, and Christopher Williams and Carl Edward Rasmussen (1996; book 2006) brought them into mainstream machine learning.

Usage note

Not to be confused with a Gaussian mixture modelGaussian mixture model, n.: A model that explains data as coming from several overlapping bell-shaped clouds, working out where each cloud sits, how spread out it is, and how likely each point is to belong to it.. Kriging and GP regression are the same mathematics under different names; kernel ridge regressionridge regression, n.: Linear regression with a built-in brake: it fits the data while also keeping the weights small, which makes predictions more stable when there are many overlapping inputs., with its penalty set to the noise variance, gives the same predictive mean without the variance.

Related

See also
  • Bayesian optimizationBayesian optimization, n.: A smart way to tune settings when each test is expensive: build a statistical guess of how settings affect results, then try whichever setting that guess says is most promising next.
  • kernel trickkernel trick, n.: A mathematical shortcut that lets simple straight-line methods find curved patterns, by computing similarities between examples as if the data had been mapped into a far richer space — without ever building that space.
  • Bayesian inferenceBayesian inference, n.: Reasoning that treats unknowns as uncertain beliefs and updates them with evidence using Bayes' theorem. The result is not one answer but a range of answers, each with a probability.
  • nonparametric modelnonparametric model, n.: A model that lets the data decide how complex it becomes: the more examples it gets, the more detail it can capture — for instance, one that predicts by comparing a new case with every stored example.
  • ridge regressionridge regression, n.: Linear regression with a built-in brake: it fits the data while also keeping the weights small, which makes predictions more stable when there are many overlapping inputs.
  • neural tangent kernelneural tangent kernel, n.: A mathematical tool showing that extremely wide neural networks learn like a simpler, well-understood method, because their internal settings barely move during training. It explains some, but not all, of deep learning's behavior.
Contrast with
  • Gaussian mixture modelGaussian mixture model, n.: A model that explains data as coming from several overlapping bell-shaped clouds, working out where each cloud sits, how spread out it is, and how likely each point is to belong to it.

Sources

  1. Rasmussen, C. E. & Williams, C. K. I. (2006). Gaussian Processes for Machine Learning. MIT Press.
  2. Williams, C. K. I. & Rasmussen, C. E. (1996). Gaussian Processes for Regression. Advances in Neural Information Processing Systems 8.

Mentioned in

regressionregression, n.: Predicting a number rather than a category — a house's price, tomorrow's temperature, a patient's blood pressure — by learning from past examples where the answer is known., Bayesian inferenceBayesian inference, n.: Reasoning that treats unknowns as uncertain beliefs and updates them with evidence using Bayes' theorem. The result is not one answer but a range of answers, each with a probability., Bayesian optimizationBayesian optimization, n.: A smart way to tune settings when each test is expensive: build a statistical guess of how settings affect results, then try whichever setting that guess says is most promising next., central limit theoremcentral limit theorem, n.: The mathematical reason bell curves are everywhere: add up many small independent random effects and the total is almost always bell-shaped, whatever the individual effects look like., hyperparameterhyperparameter, n.: A setting chosen by people before training, such as how fast a model learns or how big it is, as opposed to the internal values the model learns by itself from data., kernelkernel, n.: A word with several AI meanings: a similarity measure used by some learning methods; a small grid of numbers that scans an image inside a neural network; or a small program that runs on a GPU., kernel trickkernel trick, n.: A mathematical shortcut that lets simple straight-line methods find curved patterns, by computing similarities between examples as if the data had been mapped into a far richer space — without ever building that space., nonparametric modelnonparametric model, n.: A model that lets the data decide how complex it becomes: the more examples it gets, the more detail it can capture — for instance, one that predicts by comparing a new case with every stored example., normal distributionnormal distribution, n.: The familiar bell-shaped curve: most values cluster near the average, and values become rapidly rarer the farther they lie from it. Measurement errors and adult heights roughly follow it., priorprior, n.: What you believe, and how strongly, before looking at the evidence in front of you. In AI it also means built-in assumptions that steer what a system learns., ridge regressionridge regression, n.: Linear regression with a built-in brake: it fits the data while also keeping the weights small, which makes predictions more stable when there are many overlapping inputs., support vector machinesupport vector machine, n.: A classic method that separates two groups of examples with the widest possible gap — like laying the broadest possible road between them. A mathematical shortcut lets it draw curved boundaries too., surrogate modelsurrogate model, n.: A simpler stand-in model trained to imitate a more complex one — to explain it, or to approximate a slow or costly calculation cheaply., Bayesian neural networkBayesian neural network, n.: A neural network that keeps a whole range of plausible settings for its internal numbers instead of one best guess, so that it can say how unsure it is about each prediction., covariancecovariance, n.: A measure of whether two quantities tend to rise and fall together, move in opposite directions, or show no consistent relationship., neural tangent kernelneural tangent kernel, n.: A mathematical tool showing that extremely wide neural networks learn like a simpler, well-understood method, because their internal settings barely move during training. It explains some, but not all, of deep learning's behavior., radial basis function networkradial basis function network, n.: A three-layer network whose hidden neurons each respond to inputs near a particular reference point, like bells placed across the data; the output is a weighted mix of the bells.