probabilistic graphical model
In plain wordsA way of describing how many uncertain quantities depend on one another by drawing them as a network, which makes it practical for a computer to reason about all of them together.
A representation of a joint probability distribution over many random variables by means of a graph: nodes stand for the variables, and missing edges encode conditional independence assumptions, so the distribution factorizes into local terms over small groups of variables. The main families are directed models (Bayesian networksBayesian network, n.: A diagram of causes and effects with probabilities attached — say, how rain and a sprinkler can each make grass wet — that lets a computer update its beliefs about hidden causes when it sees evidence.), whose factors are conditional probabilities; undirected models (Markov random fieldsMarkov random field, n.: A model of many linked uncertain quantities, such as neighboring pixels in a photo, in which each one depends directly only on its neighbors, with no notion of which one causes which.), whose factors are non-negative potentials over cliques; and factor graphs, which make the factors explicit nodes. The framework separates representation, inference — computing marginals or the most probable assignment by variable elimination, belief propagationbelief propagation, n.: A way for a network of uncertain quantities to reach conclusions by having each node pass short messages to its neighbors about what it currently believes, until the beliefs settle down., junction trees, Markov chain Monte CarloMarkov chain Monte Carlo, n.: A family of methods for drawing random examples from a complicated probability distribution by taking a long, carefully designed random walk that eventually visits each region in proportion to its probability. or variational inferencevariational inference, n.: A way of approximating a hard-to-compute probability distribution by picking the closest match from a simpler family of distributions, turning a hard problem of adding up every possibility into an optimization problem. — and learning of parameters (often with the expectation–maximization algorithmexpectation–maximization algorithm, n.: A method for fitting a model when some information is hidden: guess the hidden parts using the current model, refit the model as if those guesses were right, and repeat until nothing changes.) and structure. Hidden Markov modelshidden Markov model, n.: A model of a sequence, such as speech or DNA, in which hidden states — say, the words being spoken — change step by step at random, and each state gives off signals that can be observed, such as sounds., conditional random fieldsconditional random field, n.: A statistical method for labeling every item in a sequence, such as marking each word in a sentence as a name, a place or an ordinary word, by choosing all the labels together rather than one at a time., Kalman filtersKalman filter, n.: A method that keeps a running best guess of something you cannot measure exactly, such as a vehicle's position, by blending predictions from a motion model with noisy sensor readings., mixture models and topic models are special cases.
Origin
Graphical representation of dependence goes back to Sewall Wright's path analysis (1921) and to statistical work on log-linear and Markov-field models in the 1970s. AI adopted the approach through Judea Pearl's Bayesian networks (1985–1988); Steffen Lauritzen and David Spiegelhalter's 1988 junction-tree method and Daphne Koller and Nir Friedman's 2009 textbook consolidated the unified field.
Usage note
Graphical models were a dominant paradigm of machine learning in the 1990s and 2000s. Deep generative models absorbed many of their ideas, and variational autoencodersvariational autoencoder, n.: A neural network that learns to squeeze data into a compact, slightly fuzzy code and rebuild it, arranged so that decoding random codes produces new, plausible examples. and diffusion modelsdiffusion model, n.: An AI that learns to create images, sound or video by practicing noise removal: shown pictures buried under static, it learns to clean them up, then builds new ones starting from pure static. are still drawn in graphical-model notation.
Related
- Bayesian networkBayesian network, n.: A diagram of causes and effects with probabilities attached — say, how rain and a sprinkler can each make grass wet — that lets a computer update its beliefs about hidden causes when it sees evidence.
- Markov random fieldMarkov random field, n.: A model of many linked uncertain quantities, such as neighboring pixels in a photo, in which each one depends directly only on its neighbors, with no notion of which one causes which.
- hidden Markov modelhidden Markov model, n.: A model of a sequence, such as speech or DNA, in which hidden states — say, the words being spoken — change step by step at random, and each state gives off signals that can be observed, such as sounds.
- conditional random fieldconditional random field, n.: A statistical method for labeling every item in a sequence, such as marking each word in a sentence as a name, a place or an ordinary word, by choosing all the labels together rather than one at a time.
- belief propagationbelief propagation, n.: A way for a network of uncertain quantities to reach conclusions by having each node pass short messages to its neighbors about what it currently believes, until the beliefs settle down.
- probabilistic programmingprobabilistic programming, n.: Writing a statistical model as an ordinary-looking program that includes random choices, and letting the system work out automatically which hidden values best explain the observed data.
- variational inferencevariational inference, n.: A way of approximating a hard-to-compute probability distribution by picking the closest match from a simpler family of distributions, turning a hard problem of adding up every possibility into an optimization problem.
Sources
- Koller, D. & Friedman, N. (2009). Probabilistic Graphical Models: Principles and Techniques. MIT Press.
- Lauritzen, S. L. & Spiegelhalter, D. J. (1988). Local Computations with Probabilities on Graphical Structures and Their Application to Expert Systems. Journal of the Royal Statistical Society B 50(2), 157–224.
Mentioned in
Bayesian networkBayesian network, n.: A diagram of causes and effects with probabilities attached — say, how rain and a sprinkler can each make grass wet — that lets a computer update its beliefs about hidden causes when it sees evidence., conditional random fieldconditional random field, n.: A statistical method for labeling every item in a sequence, such as marking each word in a sentence as a name, a place or an ordinary word, by choosing all the labels together rather than one at a time., hidden Markov modelhidden Markov model, n.: A model of a sequence, such as speech or DNA, in which hidden states — say, the words being spoken — change step by step at random, and each state gives off signals that can be observed, such as sounds., inference engineinference engine, n.: The part of an AI system that does the working-out: in older AI, the program that applies rules to facts; in today's AI, the software that runs a trained model to produce answers., Judea PearlJudea Pearl, prop. n.: An Israeli-American computer scientist who gave AI the mathematics for reasoning under uncertainty and, later, for telling cause and effect apart from mere correlation., linear programminglinear programming, n.: A mathematical way to find the best use of limited resources, such as a factory's machines or an airline's crews, when every cost and limit scales in simple proportion: twice the output needs twice the input., belief propagationbelief propagation, n.: A way for a network of uncertain quantities to reach conclusions by having each node pass short messages to its neighbors about what it currently believes, until the beliefs settle down., Daphne KollerDaphne Koller, prop. n.: An Israeli-American computer scientist who wrote the standard text on probabilistic reasoning in AI, co-founded the online-course company Coursera, and founded a machine-learning drug-discovery company., Markov logic networkMarkov logic network, n.: A way of combining logical rules with probabilities, so that a rule like 'friends usually like the same music' can sometimes be broken, with each rule given a weight saying how strongly it holds., Markov random fieldMarkov random field, n.: A model of many linked uncertain quantities, such as neighboring pixels in a photo, in which each one depends directly only on its neighbors, with no notion of which one causes which., Michael I. JordanMichael I. Jordan, prop. n.: An American computer scientist and statistician who did much to bring probability and statistics into machine learning, and who trained many of the field's leaders., probabilistic programmingprobabilistic programming, n.: Writing a statistical model as an ordinary-looking program that includes random choices, and letting the system work out automatically which hidden values best explain the observed data.