density estimation
In plain wordsWorking out from a sample of data how likely different values are, by drawing a smooth curve or surface showing where data points bunch up and where they are rare.
The task of inferring, from a finite sample, the probability density functionprobability density function, n.: A curve showing where a measurement is likely to fall. The area under the curve between two values gives the chance of landing between them. of the process that produced it. Parametric approaches assume a family of distributions and fit its parameters, typically by maximum likelihoodmaximum likelihood estimation, n.: Tuning a model's settings until the real data it learns from looks as likely, and so as unsurprising, to the model as possible. Most AI training, including the pretraining of chatbots, is a version of this.; a single Gaussian and the Gaussian mixture modelGaussian mixture model, n.: A model that explains data as coming from several overlapping bell-shaped clouds, working out where each cloud sits, how spread out it is, and how likely each point is to belong to it. are common choices. Nonparametric approaches let the data determine the shape: the histogram, and kernel density estimation (KDE), which places a smooth bump (kernel) on every sample and averages them, with a bandwidth h that trades spiky, high-variance estimates against oversmoothed, biased ones. Nonparametric estimates degrade quickly as dimensions grow (curse of dimensionalitycurse of dimensionality, n.: The problems that pile up when each item is described by very many measurements: even huge datasets end up thinly scattered across the possibilities, and 'near' and 'far' stop meaning much.), so high-dimensional density estimation relies on learned models: normalizing flowsnormalizing flow, n.: A generator built from a chain of reversible steps that reshape simple random noise into realistic data; because every step can be undone, it can also calculate exactly how likely any example is. and autoregressive modelsautoregressive model, n.: A model that produces a sequence one piece at a time, each new piece depending on the pieces already produced — like writing a sentence word by word without going back. give exact likelihoods, while variational autoencodersvariational autoencoder, n.: A neural network that learns to squeeze data into a compact, slightly fuzzy code and rebuild it, arranged so that decoding random codes produces new, plausible examples. and diffusion modelsdiffusion model, n.: An AI that learns to create images, sound or video by practicing noise removal: shown pictures buried under static, it learns to clean them up, then builds new ones starting from pure static. give bounds. Density estimates underpin anomaly detectionanomaly detection, n.: Automatically spotting the rare data points that don't fit the usual pattern, such as a fraudulent card payment, a machine about to fail or an intruder on a network. (low density is suspicious), generative modelinggenerative model, n.: A model that learns what a kind of data looks like — faces, sentences, songs — well enough to produce new examples that resemble the originals without copying them., Bayesian classifiers and data visualization.
The kernel density estimate of delivery times showed a second hump at 45 minutes that the average had hidden.
Origin
Karl Pearson named the histogram in the 1890s. Murray Rosenblatt (1956) and Emanuel Parzen (1962) independently introduced kernel density estimation, which is still called the Parzen–Rosenblatt window method. Bernard Silverman's 1986 book Density Estimation for Statistics and Data Analysis popularized practical bandwidth rules, and deep generative models later made density estimation a central problem of machine learning.
Related
- probability density functionprobability density function, n.: A curve showing where a measurement is likely to fall. The area under the curve between two values gives the chance of landing between them.
- Gaussian mixture modelGaussian mixture model, n.: A model that explains data as coming from several overlapping bell-shaped clouds, working out where each cloud sits, how spread out it is, and how likely each point is to belong to it.
- anomaly detectionanomaly detection, n.: Automatically spotting the rare data points that don't fit the usual pattern, such as a fraudulent card payment, a machine about to fail or an intruder on a network.
- normalizing flownormalizing flow, n.: A generator built from a chain of reversible steps that reshape simple random noise into realistic data; because every step can be undone, it can also calculate exactly how likely any example is.
- generative modelgenerative model, n.: A model that learns what a kind of data looks like — faces, sentences, songs — well enough to produce new examples that resemble the originals without copying them.
- maximum likelihood estimationmaximum likelihood estimation, n.: Tuning a model's settings until the real data it learns from looks as likely, and so as unsurprising, to the model as possible. Most AI training, including the pretraining of chatbots, is a version of this.
- curse of dimensionalitycurse of dimensionality, n.: The problems that pile up when each item is described by very many measurements: even huge datasets end up thinly scattered across the possibilities, and 'near' and 'far' stop meaning much.
Sources
- Rosenblatt, M. (1956). Remarks on Some Nonparametric Estimates of a Density Function. Annals of Mathematical Statistics 27(3).
- Parzen, E. (1962). On Estimation of a Probability Density Function and Mode. Annals of Mathematical Statistics 33(3).
- Silverman, B. W. (1986). Density Estimation for Statistics and Data Analysis. London: Chapman and Hall.
Mentioned in
anomaly detectionanomaly detection, n.: Automatically spotting the rare data points that don't fit the usual pattern, such as a fraudulent card payment, a machine about to fail or an intruder on a network., curse of dimensionalitycurse of dimensionality, n.: The problems that pile up when each item is described by very many measurements: even huge datasets end up thinly scattered across the possibilities, and 'near' and 'far' stop meaning much., DBSCANDBSCAN, n.: A clustering method that finds groups as crowded regions of data separated by sparse ones, so it can discover oddly shaped groups and set stray points aside as noise., Gaussian mixture modelGaussian mixture model, n.: A model that explains data as coming from several overlapping bell-shaped clouds, working out where each cloud sits, how spread out it is, and how likely each point is to belong to it., generative modelgenerative model, n.: A model that learns what a kind of data looks like — faces, sentences, songs — well enough to produce new examples that resemble the originals without copying them., isolation forestisolation forest, n.: A method for spotting unusual data points by repeatedly slicing the data at random: odd points get cut off from the rest after only a few slices, while ordinary points take many., nonparametric modelnonparametric model, n.: A model that lets the data decide how complex it becomes: the more examples it gets, the more detail it can capture — for instance, one that predicts by comparing a new case with every stored example., unsupervised learningunsupervised learning, n.: Machine learning without answer keys: the computer looks at unlabeled data and finds structure on its own, such as groups of similar customers or the main ways in which data vary., normalizing flownormalizing flow, n.: A generator built from a chain of reversible steps that reshape simple random noise into realistic data; because every step can be undone, it can also calculate exactly how likely any example is., probability density functionprobability density function, n.: A curve showing where a measurement is likely to fall. The area under the curve between two values gives the chance of landing between them.