Skip to content
Last Dictionary
Plane
Machine LearningMathematics & StatisticsEvaluation & Benchmarks

Bayes error rate

nounAlso: Bayes error, Bayes risk, Bayes optimal classifier, Bayes classifier, optimal Bayes error

In plain wordsThe lowest error any classifier could ever achieve on a task, because some cases are genuinely ambiguous: even a perfect model that knew the true odds would still sometimes guess wrong.

  1. The lowest achievable error rate for a given classificationclassification, n.: Sorting things into categories — spam or not spam, which animal is in a photo, which language a text is written in — by learning from examples that are already labeled. problem: the expected error of the Bayes (optimal) classifier, which assigns each input to the class with the highest true posteriorposterior, n.: What you believe after weighing new evidence against what you believed before. It is the updated belief that Bayes' theorem produces. probability. It is positive whenever classes overlap — when the same input can genuinely carry different labels because of noise, missing information or inherent ambiguity — and it depends on which features are observed, so better features can lower it while better algorithms cannot beat it. The amount by which a classifier's error exceeds the Bayes error is its excess risk, which learning theory tries to drive to zero; a method is consistent if its excess risk vanishes as data grow. The Bayes error cannot be computed for real problems, because the true distribution is unknown, but it can be bounded or estimated — for example from nearest-neighbor error rates or from disagreement among expert annotators — and it explains why some benchmarksbenchmark, n.: A fixed, shared test — a set of questions or tasks with an agreed way of scoring — used to compare AI systems with one another and to track progress over time. cannot be solved to 100%.

    Two radiologists disagree on 6% of these scans, so the Bayes error rate is probably well above zero.

R∗=Ex[ 1−max⁡yP(y∣x)]R^{*} = \mathbb{E}_{\mathbf{x}}\Big[\, 1 - \max_{y} P(y \mid \mathbf{x}) \Big]

Origin

Grew out of statistical decision theory, notably Abraham Wald's work of the 1940s, and of likelihood-based classification rules in mid-century statistics. 'Bayes error' and 'Bayes risk' were standard in pattern recognition by the early 1970s, as in Keinosuke Fukunaga's 'Introduction to Statistical Pattern Recognition' (1972); Thomas Cover and Peter Hart (1967) bounded nearest-neighbor error by twice the Bayes error.

Usage note

In regression the analogous floor is the noise variance; in language-model scaling lawsscaling laws, n.: Measured patterns showing that an AI model's error shrinks smoothly and predictably as the model is made bigger and given more data and computing power, so labs can forecast a giant model's performance from small, cheap experiments., the irreducible lossirreducible loss, n.: The part of a model's error that no amount of extra size, data or training can remove, because the thing being predicted is itself partly random — like the unpredictability of the next word in an ordinary sentence. plays a similar role. Human error rates are sometimes used as a rough proxy for the Bayes error.

Related

See also
  • classificationclassification, n.: Sorting things into categories — spam or not spam, which animal is in a photo, which language a text is written in — by learning from examples that are already labeled.
  • k-nearest neighborsk-nearest neighbors, n.: Predicting something about a new example by finding the few most similar examples already known and going with their majority answer, or their average.
  • posteriorposterior, n.: What you believe after weighing new evidence against what you believed before. It is the updated belief that Bayes' theorem produces.
  • bias–variance tradeoffbias–variance tradeoff, n.: The balancing act between models too simple to capture a pattern and models so flexible that they also learn the noise. The best predictions usually come from somewhere in between.
  • irreducible lossirreducible loss, n.: The part of a model's error that no amount of extra size, data or training can remove, because the thing being predicted is itself partly random — like the unpredictability of the next word in an ordinary sentence.
  • label noiselabel noise, n.: Mistakes in the answers attached to training or test examples — a dog photo marked 'cat' — which can mislead what a model learns and distort how good it seems.

Sources

  1. Cover, T. M. & Hart, P. E. (1967). Nearest Neighbor Pattern Classification. IEEE Transactions on Information Theory, 13(1), 21–27.
  2. Fukunaga, K. (1972; 2nd ed. 1990). Introduction to Statistical Pattern Recognition. Academic Press.