proper scoring rule
In plain wordsA way of grading probability forecasts that gives the best expected score only to a forecaster who reports what they really believe, so there is nothing to gain by exaggerating or hedging.
A rule for grading a probabilistic forecast once the outcome is known, designed so that a forecaster who expects to be graded by it can do no better on average than to state their actual beliefs; it is strictly proper when every misreport does strictly worse in expectation. Proper scoring rules make honest, calibrated probabilities the best policy, so they are the principled way to train and evaluate probabilistic models. The main examples are the logarithmic score (log losslog loss, n.: A score for probability predictions that punishes confident mistakes severely: saying '99% sure' and being wrong costs far more than saying '60% sure' and being wrong. Lower is better.), the quadratic or Brier scoreBrier score, n.: A score for probability forecasts: the average squared gap between each forecast probability and what actually happened (1 if it did, 0 if not). Zero is perfect; lower is better. and the spherical score, and, for continuous predictions, the continuous ranked probability score (CRPS) and the pinball loss for quantiles. Every strictly proper rule decomposes into calibration and resolution components, but the rules weight errors differently: the logarithmic score heavily penalizes near-zero probability on events that happen, while the Brier score is bounded. Thresholded accuracyaccuracy, n.: The share of a model's answers that are right, such as 95 out of 100. Simple and intuitive, but misleading when one kind of answer is far more common than the others., AUCAUC, abbr.: A single number from 0 to 1 summarizing how well a model ranks real positive cases above negative ones: 0.5 is no better than a coin flip and 1.0 is perfect. and the expected calibration errorexpected calibration error, n.: A single number measuring how far a model's confidence is from its actual hit rate, averaged across confidence levels; zero means its confidence can be taken at face value. are not proper: a model can improve on them while reporting less truthful probabilities. Forecasting tournaments and evaluations of language-model uncertainty rely on proper scores for this reason.
Because the contest used a proper scoring rule, shading our 70% forecasts up to 90% would only have lowered our expected score.
Origin
Glenn Brier's quadratic score (1950) and I. J. Good's logarithmic score (1952) were early examples; the computer scientist John McCarthyJohn McCarthy, prop. n.: The American computer scientist who coined the term 'artificial intelligence', invented the Lisp programming language and championed formal logic as the route to machine common sense. (1956) and Leonard Savage (1971) characterized proper scoring rules in general. Tilmann Gneiting and Adrian Raftery's 2007 review 'Strictly Proper Scoring Rules, Prediction, and Estimation' became the standard modern reference.
Related
- log losslog loss, n.: A score for probability predictions that punishes confident mistakes severely: saying '99% sure' and being wrong costs far more than saying '60% sure' and being wrong. Lower is better.
- Brier scoreBrier score, n.: A score for probability forecasts: the average squared gap between each forecast probability and what actually happened (1 if it did, 0 if not). Zero is perfect; lower is better.
- calibrationcalibration, n.: How well a model's stated confidence matches reality: when a well-calibrated system says it is 80% sure, it turns out to be right about 80% of the time.
- expected calibration errorexpected calibration error, n.: A single number measuring how far a model's confidence is from its actual hit rate, averaged across confidence levels; zero means its confidence can be taken at face value.
- cross-entropycross-entropy, n.: A score of how surprised a model's predictions are by what actually happens; lower is better. Language models are trained mainly by driving this number down on real text.
- quantile regressionquantile regression, n.: Predicting not just a typical value but a range — say, the delivery time that 90% of orders beat — by fitting models for chosen percentiles of the outcome instead of its average.
Sources
- Gneiting, T. & Raftery, A. E. (2007). Strictly Proper Scoring Rules, Prediction, and Estimation. Journal of the American Statistical Association 102(477).
- Savage, L. J. (1971). Elicitation of Personal Probabilities and Expectations. Journal of the American Statistical Association 66(336).
- McCarthy, J. (1956). Measures of the Value of Information. Proceedings of the National Academy of Sciences 42(9).
Mentioned in
Brier scoreBrier score, n.: A score for probability forecasts: the average squared gap between each forecast probability and what actually happened (1 if it did, 0 if not). Zero is perfect; lower is better., calibrationcalibration, n.: How well a model's stated confidence matches reality: when a well-calibrated system says it is 80% sure, it turns out to be right about 80% of the time., log losslog loss, n.: A score for probability predictions that punishes confident mistakes severely: saying '99% sure' and being wrong costs far more than saying '60% sure' and being wrong. Lower is better., expected calibration errorexpected calibration error, n.: A single number measuring how far a model's confidence is from its actual hit rate, averaged across confidence levels; zero means its confidence can be taken at face value.