Skip to content
Last Dictionary
Plane
Machine LearningEvaluation & Benchmarks

F1 score

nounAlso: F-score, F-measure, F1, F-beta score, Fβ score, macro-F1, micro-F1, Dice coefficient

In plain wordsA single score that combines how many of a model's alarms are right and how many real cases it catches, and stays low unless both are good.

  1. The harmonic mean of precisionprecision, n.: Of all the items a model flagged as positive, the share that really were positive; a spam filter with high precision rarely sends genuine emails to the spam folder. and recallrecall, n.: Of all the items that really were positive, the share the model managed to find; a cancer screen with high recall misses very few people who actually have cancer., ranging from 0 to 1: 2PR/(P + R), equivalently 2TP/(2TP + FP + FN). The harmonic mean is dominated by the smaller value, so a classifier cannot score well by excelling at one while neglecting the other. The general Fβ score weights recall β times as heavily as precision (F2 favors recall, F0.5 precision). For multiclass and multilabel problems, macro-F1 averages per-class scores equally, emphasizing rare classes, while micro-F1 pools all decisions and, for single-label multiclass classification, equals accuracyaccuracy, n.: The share of a model's answers that are right, such as 95 out of 100. Simple and intuitive, but misleading when one kind of answer is far more common than the others.. F1 is standard in information extraction, named-entity recognitionnamed-entity recognition, n.: Software that spots names and similar expressions in text and labels each by type: in 'Maria flew from Lagos to Paris on Monday', it marks a person, two cities and a date., question answeringquestion answering, n.: Systems that reply to people's questions with the right answer, whether by finding it in a passage they are given, searching a database or the web, or drawing on what they already know. (token-overlap F1, as in SQuADSQuAD, prop. n.: A Stanford collection of more than 100,000 questions about Wikipedia articles, each solved by highlighting the words in the article that give the answer; it drove rapid progress in machine reading from 2016.) and imbalanced classification. Criticisms are that it ignores true negatives, depends on the decision threshold and on class prevalence, and implicitly fixes the relative cost of false positives and false negatives unless β is chosen deliberately. On binary masks it equals the Dice coefficient used in image segmentation.

    Macro-F1 exposed what accuracy hid: the model never predicted the two rarest intent classes.

Fβ=(1+β2) precision⋅recallβ2⋅precision+recallF_{\beta} = (1 + \beta^2)\,\frac{\mathrm{precision} \cdot \mathrm{recall}}{\beta^2 \cdot \mathrm{precision} + \mathrm{recall}}

Origin

Derived from the effectiveness measure E that Cornelis Joost (Keith) van Rijsbergen defined in his 1979 book Information Retrieval (F = 1 − E). The name 'F-measure' arose when the score was adopted for the Fourth Message Understanding Conference (MUC-4, 1992); the letter is said to come from an unrelated function labeled F in van Rijsbergen's book. From there it spread through natural language processing and machine learning.

Usage note

Statisticians such as David Hand have argued that F1 is often misused because the effective weight it gives precision versus recall varies between classifiers; the Matthews correlation coefficientMatthews correlation coefficient, n.: A single score from −1 to +1 for a yes-or-no classifier that uses all four kinds of right and wrong answers, so it stays honest even when one answer is far more common than the other. and explicitly cost-weighted metrics are suggested alternatives.

Related

See also
  • precisionprecision, n.: Of all the items a model flagged as positive, the share that really were positive; a spam filter with high precision rarely sends genuine emails to the spam folder.
  • recallrecall, n.: Of all the items that really were positive, the share the model managed to find; a cancer screen with high recall misses very few people who actually have cancer.
  • precision–recall curveprecision–recall curve, n.: A graph showing how a model's share of correct alarms and its share of cases caught trade off as it is made more or less cautious; especially useful when positive cases are rare.
  • confusion matrixconfusion matrix, n.: A table that counts a model's right and wrong answers by type, showing for each true category how often each category was predicted, so you can see exactly which things it mixes up.
  • accuracyaccuracy, n.: The share of a model's answers that are right, such as 95 out of 100. Simple and intuitive, but misleading when one kind of answer is far more common than the others.
  • class imbalanceclass imbalance, n.: When one category is far rarer than the others in the data — say, one fraud case per thousand transactions — so a model can look accurate while missing nearly every rare case that matters.
  • Matthews correlation coefficientMatthews correlation coefficient, n.: A single score from −1 to +1 for a yes-or-no classifier that uses all four kinds of right and wrong answers, so it stays honest even when one answer is far more common than the other.

Sources

  1. van Rijsbergen, C. J. (1979). Information Retrieval, 2nd ed. London: Butterworths.
  2. Chinchor, N. (1992). MUC-4 Evaluation Metrics. Proc. Fourth Message Understanding Conference (MUC-4).
  3. Hand, D. & Christen, P. (2018). A Note on Using the F-measure for Evaluating Record Linkage Algorithms. Statistics and Computing 28.

Mentioned in

accuracyaccuracy, n.: The share of a model's answers that are right, such as 95 out of 100. Simple and intuitive, but misleading when one kind of answer is far more common than the others., classificationclassification, n.: Sorting things into categories — spam or not spam, which animal is in a photo, which language a text is written in — by learning from examples that are already labeled., class imbalanceclass imbalance, n.: When one category is far rarer than the others in the data — say, one fraud case per thousand transactions — so a model can look accurate while missing nearly every rare case that matters., confusion matrixconfusion matrix, n.: A table that counts a model's right and wrong answers by type, showing for each true category how often each category was predicted, so you can see exactly which things it mixes up., decision thresholddecision threshold, n.: The cut-off score above which a model's prediction counts as 'yes'; moving it trades missed cases against false alarms, like setting how sensitive a smoke alarm should be., exact matchexact match, n.: The simplest way to grade an AI's answer: it counts as right only if it is identical to the reference answer, after minor tidying such as ignoring capital letters., intersection over unionintersection over union, n.: A score from 0 to 1 for how well two shapes overlap, such as a predicted box and the true box around an object: the shared area divided by the combined area., multilabel classificationmultilabel classification, n.: Tagging each item with any number of labels at once — a photo can show a beach, a dog and a sunset — instead of picking just one category., named-entity recognitionnamed-entity recognition, n.: Software that spots names and similar expressions in text and labels each by type: in 'Maria flew from Lagos to Paris on Monday', it marks a person, two cities and a date., precisionprecision, n.: Of all the items a model flagged as positive, the share that really were positive; a spam filter with high precision rarely sends genuine emails to the spam folder., precision–recall curveprecision–recall curve, n.: A graph showing how a model's share of correct alarms and its share of cases caught trade off as it is made more or less cautious; especially useful when positive cases are rare., recallrecall, n.: Of all the items that really were positive, the share the model managed to find; a cancer screen with high recall misses very few people who actually have cancer., SQuADSQuAD, prop. n.: A Stanford collection of more than 100,000 questions about Wikipedia articles, each solved by highlighting the words in the article that give the answer; it drove rapid progress in machine reading from 2016., Matthews correlation coefficientMatthews correlation coefficient, n.: A single score from −1 to +1 for a yes-or-no classifier that uses all four kinds of right and wrong answers, so it stays honest even when one answer is far more common than the other., Message Understanding ConferenceMessage Understanding Conference, prop. n.: A series of US government-funded contests held from 1987 to 1998 in which research teams built programs to pull key facts, such as who did what, where and when, out of news and military messages.