Skip to content
Last Dictionary
Plane
Machine LearningMathematics & Statistics

Gaussian mixture model

nounAlso: GMM, mixture of Gaussians, mixture model, finite mixture model

In plain wordsA model that explains data as coming from several overlapping bell-shaped clouds, working out where each cloud sits, how spread out it is, and how likely each point is to belong to it.

  1. A probabilistic model in which each observation is drawn from one of K multivariate Gaussiannormal distribution, n.: The familiar bell-shaped curve: most values cluster near the average, and values become rapidly rarer the farther they lie from it. Measurement errors and adult heights roughly follow it. components, chosen with probabilities given by mixing weights; each component has its own mean vector and covariance matrix, and the overall density is the weighted sum of the component densities. Parameters are usually fitted by maximum likelihoodmaximum likelihood estimation, n.: Tuning a model's settings until the real data it learns from looks as likely, and so as unsurprising, to the model as possible. Most AI training, including the pretraining of chatbots, is a version of this. with the expectation–maximization algorithmexpectation–maximization algorithm, n.: A method for fitting a model when some information is hidden: guess the hidden parts using the current model, refit the model as if those guesses were right, and repeat until nothing changes., which alternates between computing each point's posterior responsibility for every component and re-estimating the weights, means and covariances. A fitted model serves as a soft clusteringclustering, n.: Automatically sorting things into groups of similar items when nobody has said in advance what the groups should be, like sorting a drawer of odd socks into piles by color and size. method (each point receives membership probabilities), as a density estimatordensity estimation, n.: Working out from a sample of data how likely different values are, by drawing a smooth curve or surface showing where data points bunch up and where they are rare. and as a building block of larger models such as hidden Markov modelshidden Markov model, n.: A model of a sequence, such as speech or DNA, in which hidden states — say, the words being spoken — change step by step at random, and each state gives off signals that can be observed, such as sounds. and classic speaker-recognition systems. Covariances may be full, diagonal, tied or spherical, and the number of components is often chosen with the Bayesian information criterion. Because the likelihood can diverge when a component collapses onto a single point, covariances are usually regularized.

    A two-component GMM separated the sensor readings into normal and faulty regimes and gave each reading a probability.

p(x)=∑k=1Kπk N(x∣μk,Σk),∑k=1Kπk=1p(\mathbf{x}) = \sum_{k=1}^{K} \pi_k \, \mathcal{N}(\mathbf{x} \mid \boldsymbol{\mu}_k, \boldsymbol{\Sigma}_k), \qquad \sum_{k=1}^{K} \pi_k = 1

Origin

Mixture modeling dates to Karl Pearson's 1894 analysis of shore-crab measurements gathered by Walter Weldon, which fitted a mixture of two normal distributions by the method of moments. Maximum-likelihood fitting became practical with computers, and the 1977 paper of Arthur Dempster, Nan Laird and Donald Rubin made the expectation–maximization algorithm the standard way to fit Gaussian mixtures.

Usage note

K-meansk-means clustering, n.: A simple way to split data into a chosen number of groups: guess group centers, assign every point to its nearest center, move each center to the middle of its points, and repeat. is the limiting case of a Gaussian mixture with equal spherical covariances shrinking toward zero, which turns soft responsibilities into hard assignments.

Related

See also
  • expectation–maximization algorithmexpectation–maximization algorithm, n.: A method for fitting a model when some information is hidden: guess the hidden parts using the current model, refit the model as if those guesses were right, and repeat until nothing changes.
  • clusteringclustering, n.: Automatically sorting things into groups of similar items when nobody has said in advance what the groups should be, like sorting a drawer of odd socks into piles by color and size.
  • k-means clusteringk-means clustering, n.: A simple way to split data into a chosen number of groups: guess group centers, assign every point to its nearest center, move each center to the middle of its points, and repeat.
  • density estimationdensity estimation, n.: Working out from a sample of data how likely different values are, by drawing a smooth curve or surface showing where data points bunch up and where they are rare.
  • normal distributionnormal distribution, n.: The familiar bell-shaped curve: most values cluster near the average, and values become rapidly rarer the farther they lie from it. Measurement errors and adult heights roughly follow it.
  • maximum likelihood estimationmaximum likelihood estimation, n.: Tuning a model's settings until the real data it learns from looks as likely, and so as unsurprising, to the model as possible. Most AI training, including the pretraining of chatbots, is a version of this.
  • hidden Markov modelhidden Markov model, n.: A model of a sequence, such as speech or DNA, in which hidden states — say, the words being spoken — change step by step at random, and each state gives off signals that can be observed, such as sounds.

Sources

  1. Pearson, K. (1894). Contributions to the Mathematical Theory of Evolution. Philosophical Transactions of the Royal Society of London A 185.
  2. Dempster, A. P., Laird, N. M. & Rubin, D. B. (1977). Maximum Likelihood from Incomplete Data via the EM Algorithm. Journal of the Royal Statistical Society, Series B 39(1).

Mentioned in

speech recognitionspeech recognition, n.: Technology that turns spoken words into written text, as when a phone transcribes a voice message or captions appear automatically under a video., clusteringclustering, n.: Automatically sorting things into groups of similar items when nobody has said in advance what the groups should be, like sorting a drawer of odd socks into piles by color and size., density estimationdensity estimation, n.: Working out from a sample of data how likely different values are, by drawing a smooth curve or surface showing where data points bunch up and where they are rare., expectation–maximization algorithmexpectation–maximization algorithm, n.: A method for fitting a model when some information is hidden: guess the hidden parts using the current model, refit the model as if those guesses were right, and repeat until nothing changes., Gaussian processGaussian process, n.: A way of predicting an unknown curve or surface that gives not just a best guess at each point but an honest error bar, narrow near measured points and wide far from them., generative modelgenerative model, n.: A model that learns what a kind of data looks like — faces, sentences, songs — well enough to produce new examples that resemble the originals without copying them., k-means clusteringk-means clustering, n.: A simple way to split data into a chosen number of groups: guess group centers, assign every point to its nearest center, move each center to the middle of its points, and repeat., Bayesian information criterionBayesian information criterion, n.: A score for comparing statistical models that, like AIC, rewards good fit and penalizes extra adjustable numbers, but penalizes them more as the dataset grows, favoring simpler models. Lower is better., MFCCMFCC, abbr.: A short list of numbers, usually about 13, that summarizes the shape of a brief slice of sound in a way that mimics human hearing; for decades the standard input to speech recognizers.