Skip to content
Last Dictionary
Plane
Machine LearningEvaluation & Benchmarks

accuracy

nounAlso: classification accuracy, top-k accuracy, top-1 accuracy, top-5 accuracy, balanced accuracy, error rate

In plain wordsThe share of a model's answers that are right, such as 95 out of 100. Simple and intuitive, but misleading when one kind of answer is far more common than the others.

  1. 1

    Classification

    The proportion of predictions that match the true labels: correct predictions divided by all predictions, or (TP + TN)/(TP + TN + FP + FN) in a binary confusion matrixconfusion matrix, n.: A table that counts a model's right and wrong answers by type, showing for each true category how often each category was predicted, so you can see exactly which things it mixes up.; its complement is the error rate. Accuracy is the default headline metric for balanced classificationclassification, n.: Sorting things into categories — spam or not spam, which animal is in a photo, which language a text is written in — by learning from examples that are already labeled. problems and for many benchmarksbenchmark, n.: A fixed, shared test — a set of questions or tasks with an agreed way of scoring — used to compare AI systems with one another and to track progress over time., often computed as exact matchexact match, n.: The simplest way to grade an AI's answer: it counts as right only if it is identical to the reference answer, after minor tidying such as ignoring capital letters. on multiple-choice or short answers. It misleads under class imbalanceclass imbalance, n.: When one category is far rarer than the others in the data — say, one fraud case per thousand transactions — so a model can look accurate while missing nearly every rare case that matters.: a fraud model that labels every transaction legitimate is 99.9% accurate if only 0.1% are fraudulent. Balanced accuracy, the mean of per-class recallrecall, n.: Of all the items that really were positive, the share the model managed to find; a cancer screen with high recall misses very few people who actually have cancer., corrects for this, and precisionprecision, n.: Of all the items a model flagged as positive, the share that really were positive; a spam filter with high precision rarely sends genuine emails to the spam folder., recall, the F1 scoreF1 score, n.: A single score that combines how many of a model's alarms are right and how many real cases it catches, and stays low unless both are good. and ROCROC curve, n.: A graph showing how a detector's hit rate and false-alarm rate change together as it is made more or less trigger-happy; the closer the curve hugs the top-left corner, the better. analysis give a fuller picture. Accuracy also depends on the decision threshold and ignores confidence, so an accurate model can be poorly calibrated (calibrationcalibration, n.: How well a model's stated confidence matches reality: when a well-calibrated system says it is 80% sure, it turns out to be right about 80% of the time.). Top-k accuracy counts a prediction as correct if the true label is among the k highest-scoring classes, as in ImageNet's top-5 accuracy.

    The model hit 98% accuracy, but only because 98% of the emails weren't spam.

  2. 2

    General usage

    Loosely, the overall correctness of an AI system's outputs, as in news coverage of 'the accuracy of chatbot answers', usually without a defined metric. In metrology, accuracy describes how near a measurement lies to the true value, whereas precision describes how tightly repeated measurements cluster, however far they sit from the truth.

Accuracy=TP+TNTP+TN+FP+FN\mathrm{Accuracy} = \frac{TP + TN}{TP + TN + FP + FN}

Origin

The everyday word is centuries old; as the standard summary of classifier performance it was established in statistical pattern recognition by the 1950s and 1960s. Top-5 accuracy became famous through the ImageNet Large Scale Visual Recognition Challenge (2010–2017).

Usage note

The machine-learning metric called precisionprecision, n.: Of all the items a model flagged as positive, the share that really were positive; a spam filter with high precision rarely sends genuine emails to the spam folder. is not the measurement-science sense of precision; in classification, accuracy and precision answer different questions.

Related

See also
  • confusion matrixconfusion matrix, n.: A table that counts a model's right and wrong answers by type, showing for each true category how often each category was predicted, so you can see exactly which things it mixes up.
  • precisionprecision, n.: Of all the items a model flagged as positive, the share that really were positive; a spam filter with high precision rarely sends genuine emails to the spam folder.
  • recallrecall, n.: Of all the items that really were positive, the share the model managed to find; a cancer screen with high recall misses very few people who actually have cancer.
  • F1 scoreF1 score, n.: A single score that combines how many of a model's alarms are right and how many real cases it catches, and stays low unless both are good.
  • class imbalanceclass imbalance, n.: When one category is far rarer than the others in the data — say, one fraud case per thousand transactions — so a model can look accurate while missing nearly every rare case that matters.
  • calibrationcalibration, n.: How well a model's stated confidence matches reality: when a well-calibrated system says it is 80% sure, it turns out to be right about 80% of the time.
  • exact matchexact match, n.: The simplest way to grade an AI's answer: it counts as right only if it is identical to the reference answer, after minor tidying such as ignoring capital letters.
  • baselinebaseline, n.: A simple reference method that a new model must beat to show it is worth anything, such as always guessing the most common answer or reusing yesterday's value.

Sources

  1. Sokolova, M. & Lapalme, G. (2009). A Systematic Analysis of Performance Measures for Classification Tasks. Information Processing & Management 45(4).

Mentioned in

benchmarkbenchmark, n.: A fixed, shared test — a set of questions or tasks with an agreed way of scoring — used to compare AI systems with one another and to track progress over time., classificationclassification, n.: Sorting things into categories — spam or not spam, which animal is in a photo, which language a text is written in — by learning from examples that are already labeled., anomaly detectionanomaly detection, n.: Automatically spotting the rare data points that don't fit the usual pattern, such as a fraudulent card payment, a machine about to fail or an intruder on a network., class imbalanceclass imbalance, n.: When one category is far rarer than the others in the data — say, one fraud case per thousand transactions — so a model can look accurate while missing nearly every rare case that matters., confusion matrixconfusion matrix, n.: A table that counts a model's right and wrong answers by type, showing for each true category how often each category was predicted, so you can see exactly which things it mixes up., exact matchexact match, n.: The simplest way to grade an AI's answer: it counts as right only if it is identical to the reference answer, after minor tidying such as ignoring capital letters., F1 scoreF1 score, n.: A single score that combines how many of a model's alarms are right and how many real cases it catches, and stays low unless both are good., mean average precisionmean average precision, n.: The standard score for object detectors: it rewards finding all the objects while raising few false alarms, averaged over every type of object. Higher is better, up to 100., MMLUMMLU, prop. n.: A giant multiple-choice exam for AI covering 57 subjects, from school math and history to law and medicine; for years it was the most quoted test of a model's general knowledge., precisionprecision, n.: Of all the items a model flagged as positive, the share that really were positive; a spam filter with high precision rarely sends genuine emails to the spam folder., predictionprediction, n.: A model's output for a given input: its best guess at an unknown answer. In AI it need not be about the future; labeling a photo 'cat' counts as a prediction., Cohen's kappaCohen's kappa, n.: A score for how much two people labeling the same items agree beyond what lucky guessing would produce: 1 means perfect agreement, 0 means no better than chance., expected calibration errorexpected calibration error, n.: A single number measuring how far a model's confidence is from its actual hit rate, averaged across confidence levels; zero means its confidence can be taken at face value., Matthews correlation coefficientMatthews correlation coefficient, n.: A single score from −1 to +1 for a yes-or-no classifier that uses all four kinds of right and wrong answers, so it stays honest even when one answer is far more common than the other., proper scoring ruleproper scoring rule, n.: A way of grading probability forecasts that gives the best expected score only to a forecaster who reports what they really believe, so there is nothing to gain by exaggerating or hedging.