Cohen's kappa
In plain wordsA score for how much two people labeling the same items agree beyond what lucky guessing would produce: 1 means perfect agreement, 0 means no better than chance.
A chance-corrected index of how often two raters put the same items into the same categories: it expresses how far their observed agreement exceeds the agreement their individual labeling rates would produce by luck, as a fraction of the largest excess possible. Observed agreement p_o is the share of items labeled identically; expected agreement p_e is computed from each rater's own category frequencies, as if both had labeled at random with those frequencies. Kappa is 1 for perfect agreement, 0 for chance-level agreement and negative for systematic disagreement. Weighted kappa (1968) gives partial credit for near misses on ordered scales. Widely quoted bands from J. Richard Landis and Gary Koch (1977) call 0.41–0.60 'moderate' and 0.61–0.80 'substantial', but these cut-offs are conventions without empirical grounding. Kappa depends on how common each category is: with highly skewed classes it can be low even when raw agreement is high.
Kappa between the two clinicians was 0.72, substantial by the usual reading.
Origin
Introduced by the American psychologist Jacob Cohen in 'A Coefficient of Agreement for Nominal Scales' (Educational and Psychological Measurement, 1960), building on William Scott's pi (1955). Cohen added weighted kappa in 1968, and Joseph Fleiss extended chance-corrected agreement to many raters in 1971.
Usage note
For more than two raters, use Fleiss' kappa or Krippendorff's alpha. Because kappa is sensitive to class balance, reports should also give raw agreement and the label distribution.
Related
- inter-annotator agreementinter-annotator agreement, n.: A measure of how often different people labeling the same data give the same answer. Low agreement warns that the task is unclear or the labels are unreliable.
- annotationannotation, n.: Adding information to raw data by hand or with software — marking objects in photos, tagging names in sentences, transcribing speech or ranking chatbot answers — so machines can learn from it.
- label noiselabel noise, n.: Mistakes in the answers attached to training or test examples — a dog photo marked 'cat' — which can mislead what a model learns and distort how good it seems.
- confusion matrixconfusion matrix, n.: A table that counts a model's right and wrong answers by type, showing for each true category how often each category was predicted, so you can see exactly which things it mixes up.
- accuracyaccuracy, n.: The share of a model's answers that are right, such as 95 out of 100. Simple and intuitive, but misleading when one kind of answer is far more common than the others.
Sources
- Cohen, J. (1960). A Coefficient of Agreement for Nominal Scales. Educational and Psychological Measurement 20(1), 37–46.
- Landis, J. R. & Koch, G. G. (1977). The Measurement of Observer Agreement for Categorical Data. Biometrics 33(1).
Mentioned in
inter-annotator agreementinter-annotator agreement, n.: A measure of how often different people labeling the same data give the same answer. Low agreement warns that the task is unclear or the labels are unreliable.