Skip to content
Last Dictionary
Plane
Machine LearningMathematics & StatisticsEvaluation & BenchmarksSafety & Alignment

conformal prediction

nounAlso: conformal inference, split conformal prediction, inductive conformal prediction, conformalized quantile regression, prediction set, nonconformity score

In plain wordsA way to wrap any prediction model so that, instead of one guess, it gives a range or set of answers that contains the truth a chosen share of the time, such as 90%.

  1. A framework for turning the outputs of any predictive model into prediction sets (for classification) or intervals (for regression) with a guaranteed coverage rate: for a chosen error level α, the set contains the true answer with probability at least 1 − α, assuming only that the data are exchangeable, as independent and identically distributed data are. In the widely used split (inductive) version, a held-out calibration set is scored with a nonconformity score — such as the absolute residual, or one minus the probability given to the true class — and a quantile of those scores, with a small finite-sample correction, sets the threshold for new predictions. The guarantee holds for any model and sample size but is marginal, averaged over inputs rather than conditional on each one, and it is no longer guaranteed under distribution shiftdistribution shift, n.: When the data an AI system meets in real use differs from the data it was trained on — new users, places, times or habits — often making it perform worse than tests suggested. that breaks exchangeability. Adaptive scores give harder inputs larger sets, a practical uncertainty signal; applications include medical triage, quantile regressionquantile regression, n.: Predicting not just a typical value but a range — say, the delivery time that 90% of orders beat — by fitting models for chosen percentiles of the outcome instead of its average. intervals and calibrated abstentionabstention, n.: When an AI declines to answer because it is not sure enough — saying 'I don't know' or asking for more information — instead of guessing and risking a confident mistake. or fact filtering for language modelslarge language model, n.: A computer program trained on enormous amounts of text to predict the words that come next. At vast scale, that simple skill lets it chat, answer questions, translate, summarize and write code..

    With split conformal prediction at α = 0.1, the classifier returns one label for easy scans and three for ambiguous ones.

Pr⁡(Yn+1∈C(Xn+1))≥1−α\Pr\big(Y_{n+1} \in C(X_{n+1})\big) \ge 1 - \alpha

Origin

Introduced by Alexander Gammerman, Vladimir Vovk and Vladimir Vapnik in 1998 ('learning by transduction'), followed by a p-value formulation from Craig Saunders and colleagues (1999) and the split variant, first called inductive confidence machines, by Harris Papadopoulos and colleagues (2002). Vovk, Gammerman and Glenn Shafer's 'Algorithmic Learning in a Random World' (2005) systematized it, and work by Jing Lei, Larry Wasserman and others in the 2010s spread it through statistics.

Usage note

A 90% guarantee is an average over the data distribution: sets can systematically miss for particular subgroups. The name refers to how well a new example 'conforms' to earlier ones, not to conformal maps in geometry.

Related

See also
  • calibrationcalibration, n.: How well a model's stated confidence matches reality: when a well-calibrated system says it is 80% sure, it turns out to be right about 80% of the time.
  • quantile regressionquantile regression, n.: Predicting not just a typical value but a range — say, the delivery time that 90% of orders beat — by fitting models for chosen percentiles of the outcome instead of its average.
  • abstentionabstention, n.: When an AI declines to answer because it is not sure enough — saying 'I don't know' or asking for more information — instead of guessing and risking a confident mistake.
  • distribution shiftdistribution shift, n.: When the data an AI system meets in real use differs from the data it was trained on — new users, places, times or habits — often making it perform worse than tests suggested.
  • confidence intervalconfidence interval, n.: A range of plausible values for something being estimated, such as 'accuracy 82 percent, give or take 2 points', reflecting the uncertainty that comes from measuring only a sample.
  • bootstrapbootstrap, n.: Estimating how much a result might vary by repeatedly re-drawing samples, with replacement, from the data you already have — like re-running a study many times without collecting new data.
Contrast with
  • calibrationcalibration, n.: How well a model's stated confidence matches reality: when a well-calibrated system says it is 80% sure, it turns out to be right about 80% of the time.

Sources

  1. Vovk, V., Gammerman, A. & Shafer, G. (2005). Algorithmic Learning in a Random World. Springer.
  2. Angelopoulos, A. N. & Bates, S. (2021). A Gentle Introduction to Conformal Prediction and Distribution-Free Uncertainty Quantification. arXiv:2107.07511.

Mentioned in

calibrationcalibration, n.: How well a model's stated confidence matches reality: when a well-calibrated system says it is 80% sure, it turns out to be right about 80% of the time., quantile regressionquantile regression, n.: Predicting not just a typical value but a range — say, the delivery time that 90% of orders beat — by fitting models for chosen percentiles of the outcome instead of its average., uncertainty quantificationuncertainty quantification, n.: Measuring how unsure an AI system's predictions are, and why, so that people know when to trust an answer and when to double-check it.