likelihood
In plain wordsA score for a possible explanation of some data: how probable that explanation, or that setting of a model, would have made the data actually seen.
The probability (or densityprobability density function, n.: A curve showing where a measurement is likely to fall. The area under the curve between two values gives the chance of landing between them.) of the observed data regarded as a function of the model's parametersparameter, n.: One of the adjustable numbers inside an AI model that are tuned during training. Large language models have billions of them, and together they store what the model has learned. or hypothesis, L(θ) = p(data | θ), with the data held fixed. It is not a probability distribution over θ — it need not sum or integrate to one across parameter values — but it ranks parameter values by how well they predict what was seen. Maximum likelihood estimationmaximum likelihood estimation, n.: Tuning a model's settings until the real data it learns from looks as likely, and so as unsurprising, to the model as possible. Most AI training, including the pretraining of chatbots, is a version of this. picks the parameters that maximize it; Bayesian inferenceBayesian inference, n.: Reasoning that treats unknowns as uncertain beliefs and updates them with evidence using Bayes' theorem. The result is not one answer but a range of answers, each with a probability. multiplies it by a priorprior, n.: What you believe, and how strongly, before looking at the evidence in front of you. In AI it also means built-in assumptions that steer what a system learns. to obtain a posteriorposterior, n.: What you believe after weighing new evidence against what you believed before. It is the updated belief that Bayes' theorem produces.; and the marginal likelihood, its average over the prior, scores whole models. For independent data the likelihood is a product over examples, so in practice one works with the log-likelihoodlog-likelihood, n.: The logarithm of how probable a model finds the observed data. Taking logs turns a product of many tiny probabilities into a manageable sum; training usually minimizes its negative.. Training a language modellarge language model, n.: A computer program trained on enormous amounts of text to predict the words that come next. At vast scale, that simple skill lets it chat, answer questions, translate, summarize and write code. with cross-entropycross-entropy, n.: A score of how surprised a model's predictions are by what actually happens; lower is better. Language models are trained mainly by driving this number down on real text. maximizes the likelihood of its training text.
Under the fitted model the held-out data have a much lower likelihood, a sign of overfitting.
Origin
Ronald Fisher gave the word its technical meaning in a 1921 paper on the correlation coefficient, deliberately separating likelihood from probability, and built maximum likelihood estimation on it in 1922. Harold Jeffreys and later Bayesians adopted it as the data's contribution to Bayes' theorem. Before Fisher, 'likelihood' was an everyday synonym for probability.
Usage note
In everyday English 'likelihood' means probability. In statistics, the likelihood of a hypothesis is the probability of the data under it, which differs from the probability of the hypothesis itself (its posteriorposterior, n.: What you believe after weighing new evidence against what you believed before. It is the updated belief that Bayes' theorem produces.).
Related
- log-likelihoodlog-likelihood, n.: The logarithm of how probable a model finds the observed data. Taking logs turns a product of many tiny probabilities into a manageable sum; training usually minimizes its negative.
- maximum likelihood estimationmaximum likelihood estimation, n.: Tuning a model's settings until the real data it learns from looks as likely, and so as unsurprising, to the model as possible. Most AI training, including the pretraining of chatbots, is a version of this.
- Bayes' theoremBayes' theorem, n.: A formula for updating how much you believe something after seeing new evidence: weigh your earlier belief by how well that explanation predicts what you actually observed.
- priorprior, n.: What you believe, and how strongly, before looking at the evidence in front of you. In AI it also means built-in assumptions that steer what a system learns.
- posteriorposterior, n.: What you believe after weighing new evidence against what you believed before. It is the updated belief that Bayes' theorem produces.
- cross-entropycross-entropy, n.: A score of how surprised a model's predictions are by what actually happens; lower is better. Language models are trained mainly by driving this number down on real text.
- posteriorposterior, n.: What you believe after weighing new evidence against what you believed before. It is the updated belief that Bayes' theorem produces.
Mentioned in
probabilityprobability, n.: How likely something is, on a scale from 0 for the impossible to 1 for the certain; rolling a six with a fair die scores one-sixth. AI systems state their predictions, even each next word, as probabilities., Bayes' theoremBayes' theorem, n.: A formula for updating how much you believe something after seeing new evidence: weigh your earlier belief by how well that explanation predicts what you actually observed., Bayesian inferenceBayesian inference, n.: Reasoning that treats unknowns as uncertain beliefs and updates them with evidence using Bayes' theorem. The result is not one answer but a range of answers, each with a probability., log-likelihoodlog-likelihood, n.: The logarithm of how probable a model finds the observed data. Taking logs turns a product of many tiny probabilities into a manageable sum; training usually minimizes its negative., maximum likelihood estimationmaximum likelihood estimation, n.: Tuning a model's settings until the real data it learns from looks as likely, and so as unsurprising, to the model as possible. Most AI training, including the pretraining of chatbots, is a version of this., Occam's razorOccam's razor, n.: A rule of thumb for choosing between explanations: if two fit the evidence equally well, bet on the one that assumes less. In AI it motivates preferring simpler models, which are less likely to fit noise., posteriorposterior, n.: What you believe after weighing new evidence against what you believed before. It is the updated belief that Bayes' theorem produces., priorprior, n.: What you believe, and how strongly, before looking at the evidence in front of you. In AI it also means built-in assumptions that steer what a system learns., Akaike information criterionAkaike information criterion, n.: A score for comparing statistical models that rewards fitting the data well but charges a penalty for every extra adjustable number, to avoid picking needlessly complicated models. Lower is better., Bayesian information criterionBayesian information criterion, n.: A score for comparing statistical models that, like AIC, rewards good fit and penalizes extra adjustable numbers, but penalizes them more as the dataset grows, favoring simpler models. Lower is better., evidence lower boundevidence lower bound, n.: A quantity that is always less than or equal to how well a model explains its data, but much easier to calculate; pushing it upward is how models such as VAEs learn., maximum a posteriori estimationmaximum a posteriori estimation, n.: Picking the single most believable setting of a model after combining the evidence with what was assumed beforehand. In practice it often amounts to ordinary training plus a penalty that keeps settings modest., probability density functionprobability density function, n.: A curve showing where a measurement is likely to fall. The area under the curve between two values gives the chance of landing between them.