calibration
In plain wordsHow well a model's stated confidence matches reality: when a well-calibrated system says it is 80% sure, it turns out to be right about 80% of the time.
- 1
Probability calibration
Agreement between a model's predicted probabilities and observed frequencies: among all cases assigned probability p of an event, the event should occur in a fraction close to p. Calibration is distinct from discrimination, the ability to rank positives above negatives measured by AUCAUC, abbr.: A single number from 0 to 1 summarizing how well a model ranks real positive cases above negative ones: 0.5 is no better than a coin flip and 1.0 is perfect.: a model can rank perfectly yet be overconfident. It is assessed with reliability diagrams, which bin predictions by confidence and plot observed accuracy against mean confidence; with summary measures such as the expected calibration errorexpected calibration error, n.: A single number measuring how far a model's confidence is from its actual hit rate, averaged across confidence levels; zero means its confidence can be taken at face value.; and with proper scoring rulesproper scoring rule, n.: A way of grading probability forecasts that gives the best expected score only to a forecaster who reports what they really believe, so there is nothing to gain by exaggerating or hedging. such as log losslog loss, n.: A score for probability predictions that punishes confident mistakes severely: saying '99% sure' and being wrong costs far more than saying '60% sure' and being wrong. Lower is better. and the Brier scoreBrier score, n.: A score for probability forecasts: the average squared gap between each forecast probability and what actually happened (1 if it did, 0 if not). Zero is perfect; lower is better., which reward calibration and discrimination together. Post-hoc recalibration fits a mapping from scores to probabilities on held-out data: Platt scaling (a logistic fit), isotonic regression (a monotone step function) or temperature scaling (dividing a network's logitslogit, n.: A raw score that a model gives each possible answer before the scores are turned into probabilities. In statistics, the 'log-odds': a probability rewritten on a scale running from minus to plus infinity. by one learned constant; see temperaturetemperature, n.: A setting that controls how adventurous an AI model's word choices are. Low temperature makes it pick the safest, most likely words; high temperature makes it more varied and surprising, but more error-prone.). Modern deep networks are often overconfident. For language modelslarge language model, n.: A computer program trained on enormous amounts of text to predict the words that come next. At vast scale, that simple skill lets it chat, answer questions, translate, summarize and write code., calibrated answer probabilities and verbalized confidence underpin abstentionabstention, n.: When an AI declines to answer because it is not sure enough — saying 'I don't know' or asking for more information — instead of guessing and risking a confident mistake. and selective answering and help reduce the harm of hallucinationshallucination, n.: When an AI states something false or made up — a fake fact, quotation or source — fluently and confidently, as if it were true..
After temperature scaling, the classifier's 90%-confidence predictions were right 89% of the time instead of 71%.
- 2
Fairness
In algorithmic fairnessalgorithmic fairness, n.: The effort to make automated decisions fair: to say in numbers what fair treatment means, measure whether a system achieves it, and correct the system if it does not., the requirement that a risk score be equally well calibrated within each protected group, so that a given score means the same probability of the outcome whatever a person's group. Results by Jon Kleinberg and colleagues and by Alexandra Chouldechova (2016–2017) showed that when base rates differ between groups, calibration within groups generally cannot coexist with equal false positive and false negative rates (fairness impossibility theoremfairness impossibility theorem, n.: A mathematical result showing that when groups differ in how often an outcome really occurs, no imperfect prediction system can be fair by every common measure at once.), a tension at the heart of the COMPASCOMPAS, prop. n.: Commercial software used in parts of the US justice system to rate how likely someone is to commit another crime; a 2016 news investigation asking whether it was unfair to Black defendants made it famous. debate.
- 3
Vision and robotics
Estimation of a camera's or sensor's internal parameters (focal length, lens distortion) and external parameters (position and orientation), so that measurements can be mapped accurately to the physical world; a prerequisite for 3D reconstruction3D reconstruction, n.: Building a 3D model of an object or place, its shape and often its colors, from photos, video or depth-sensor scans., sensor fusionsensor fusion, n.: Combining readings from several different sensors, such as cameras, radar, lidar, GPS and motion sensors, to get a more accurate and reliable picture than any one sensor gives alone. and robot perception.
Origin
Calibration of probability forecasts was studied first in meteorology, where Glenn Brier (1950) and later Allan Murphy and Robert Winkler developed scores and reliability diagrams; A. Philip Dawid analyzed it theoretically in 'The Well-Calibrated Bayesian' (1982). In machine learning, John Platt's sigmoid fitting (1999) and Bianca Zadrozny and Charles Elkan's binning and isotonic-regression methods (2001–2002) recalibrated classifiers, and Chuan Guo and colleagues' 2017 study of overconfident neural networks introduced temperature scaling.
Usage note
Saurav Kadavath and colleagues (2022) found large pretrained language models fairly well calibrated on multiple-choice questions, and the GPT-4 technical report (2023) noted that post-training reduced calibration. Calibration is not usefulness: always predicting the base rate is perfectly calibrated.
Related
- expected calibration errorexpected calibration error, n.: A single number measuring how far a model's confidence is from its actual hit rate, averaged across confidence levels; zero means its confidence can be taken at face value.
- Brier scoreBrier score, n.: A score for probability forecasts: the average squared gap between each forecast probability and what actually happened (1 if it did, 0 if not). Zero is perfect; lower is better.
- log losslog loss, n.: A score for probability predictions that punishes confident mistakes severely: saying '99% sure' and being wrong costs far more than saying '60% sure' and being wrong. Lower is better.
- proper scoring ruleproper scoring rule, n.: A way of grading probability forecasts that gives the best expected score only to a forecaster who reports what they really believe, so there is nothing to gain by exaggerating or hedging.
- temperaturetemperature, n.: A setting that controls how adventurous an AI model's word choices are. Low temperature makes it pick the safest, most likely words; high temperature makes it more varied and surprising, but more error-prone.
- abstentionabstention, n.: When an AI declines to answer because it is not sure enough — saying 'I don't know' or asking for more information — instead of guessing and risking a confident mistake.
- fairness impossibility theoremfairness impossibility theorem, n.: A mathematical result showing that when groups differ in how often an outcome really occurs, no imperfect prediction system can be fair by every common measure at once.
- conformal predictionconformal prediction, n.: A way to wrap any prediction model so that, instead of one guess, it gives a range or set of answers that contains the truth a chosen share of the time, such as 90%.
- AUCAUC, abbr.: A single number from 0 to 1 summarizing how well a model ranks real positive cases above negative ones: 0.5 is no better than a coin flip and 1.0 is perfect.
Sources
- Guo, C., Pleiss, G., Sun, Y. & Weinberger, K. Q. (2017). On Calibration of Modern Neural Networks. Proc. ICML 2017.
- Dawid, A. P. (1982). The Well-Calibrated Bayesian. Journal of the American Statistical Association 77(379).
- Kleinberg, J., Mullainathan, S. & Raghavan, M. (2017). Inherent Trade-Offs in the Fair Determination of Risk Scores. Proc. 8th Innovations in Theoretical Computer Science Conference (ITCS).
Mentioned in
accuracyaccuracy, n.: The share of a model's answers that are right, such as 95 out of 100. Simple and intuitive, but misleading when one kind of answer is far more common than the others., classificationclassification, n.: Sorting things into categories — spam or not spam, which animal is in a photo, which language a text is written in — by learning from examples that are already labeled., classifierclassifier, n.: A program or model that sorts inputs into categories, for example flagging an email as spam or a chatbot message as harmful. Many AI safety filters are classifiers., hallucinationhallucination, n.: When an AI states something false or made up — a fake fact, quotation or source — fluently and confidently, as if it were true., Humanity's Last ExamHumanity's Last Exam, prop. n.: A test of 2,500 extremely hard questions with definite answers, written by about a thousand experts worldwide and billed as the last academic exam AI would need; at launch, top models scored under 10%., probabilityprobability, n.: How likely something is, on a scale from 0 for the impossible to 1 for the certain; rolling a six with a fair die scores one-sixth. AI systems state their predictions, even each next word, as probabilities., temperaturetemperature, n.: A setting that controls how adventurous an AI model's word choices are. Low temperature makes it pick the safest, most likely words; high temperature makes it more varied and surprising, but more error-prone., abstentionabstention, n.: When an AI declines to answer because it is not sure enough — saying 'I don't know' or asking for more information — instead of guessing and risking a confident mistake., AUCAUC, abbr.: A single number from 0 to 1 summarizing how well a model ranks real positive cases above negative ones: 0.5 is no better than a coin flip and 1.0 is perfect., base modelbase model, n.: A language model fresh from its first broad training on text, before being taught to follow instructions or chat. It continues whatever text it is given instead of answering., BIG-benchBIG-bench, prop. n.: A huge collaborative test of 204 tasks, written by hundreds of researchers, built to probe what language models can and cannot do as they grow larger., Brier scoreBrier score, n.: A score for probability forecasts: the average squared gap between each forecast probability and what actually happened (1 if it did, 0 if not). Zero is perfect; lower is better., BrowseCompBrowseComp, prop. n.: A test of AI web-research agents: 1,266 puzzle-like questions whose answers take a long, persistent online hunt to uncover, though each answer is short and simple to confirm., class imbalanceclass imbalance, n.: When one category is far rarer than the others in the data — say, one fraud case per thousand transactions — so a model can look accurate while missing nearly every rare case that matters., click-through rate predictionclick-through rate prediction, n.: Estimating how likely a particular person is to click on a particular ad, search result or recommended item: a prediction made billions of times a day to decide what each user is shown., confabulationconfabulation, n.: When an AI fills a gap in its knowledge with an invented but plausible-sounding answer, much as some people with memory disorders sincerely 'remember' things that never happened., decision thresholddecision threshold, n.: The cut-off score above which a model's prediction counts as 'yes'; moving it trades missed cases against false alarms, like setting how sensitive a smoke alarm should be., factualityfactuality, n.: How far what an AI says is actually true about the world. A factual model gets facts, numbers, names and dates right and does not make things up., fairness impossibility theoremfairness impossibility theorem, n.: A mathematical result showing that when groups differ in how often an outcome really occurs, no imperfect prediction system can be fair by every common measure at once., focal lossfocal loss, n.: A way of scoring a model's mistakes during training that pays less attention to examples it already gets right easily, so that learning concentrates on the hard and rare cases., HELMHELM, prop. n.: A large, open testing framework from Stanford that grades many language models on many kinds of tasks and qualities — not only accuracy but also fairness, toxicity and efficiency., honestyhonesty, n.: An AI's disposition to say only what it actually believes, to admit uncertainty, and never to try to mislead — even when a falsehood would please or persuade., log losslog loss, n.: A score for probability predictions that punishes confident mistakes severely: saying '99% sure' and being wrong costs far more than saying '60% sure' and being wrong. Lower is better., log probabilitylog probability, n.: The logarithm of the probability a model gives to a token or a text. Computers use it because multiplying many tiny probabilities runs into rounding problems, whereas adding their logarithms does not.