inter-annotator agreement
In plain wordsA measure of how often different people labeling the same data give the same answer. Low agreement warns that the task is unclear or the labels are unreliable.
A measure of how consistently annotators working separately give the same items the same labelslabel, n.: The correct answer attached to an example in a dataset — 'cat' on a photo of a cat, 'positive' on an upbeat review — that a learning system is trained to predict., used to judge whether an annotation scheme is well defined and whether its labels can serve as ground truthground truth, n.: The reference answers treated as correct when checking or training an AI system — for example, the confirmed diagnoses that a medical-imaging model's guesses are compared against.. Raw percent agreement overstates reliability because some agreement happens by chance, so chance-corrected coefficients are preferred: Cohen's kappaCohen's kappa, n.: A score for how much two people labeling the same items agree beyond what lucky guessing would produce: 1 means perfect agreement, 0 means no better than chance. for two annotators, Fleiss' kappa for a fixed number of raters per item, and Krippendorff's alpha, which handles many annotators, missing judgments and ordinal or interval scales. For continuous scores, correlation or intraclass correlation is used. Agreement also sets a practical ceiling on evaluation: a model's score against references cannot be interpreted more finely than the references' own consistency, so careful dataset papers report it.
Agreement was only 0.41 kappa, so we merged the two most confusable categories.
Origin
Reliability coefficients for human judges come from psychology and content analysis: William Scott's pi (1955), Jacob Cohen's kappa (1960) and Klaus Krippendorff's alpha (c. 1970). Jean Carletta's 1996 article in Computational Linguistics persuaded language researchers to report kappa, and 'inter-annotator agreement' became the field's standard term.
Usage note
Agreement is not accuracy: annotators can agree on a wrong answer. Conversely, some disagreement reflects genuinely ambiguous or subjective items rather than error, and some researchers now keep the full distribution of judgments instead of forcing a single label.
Related
- Cohen's kappaCohen's kappa, n.: A score for how much two people labeling the same items agree beyond what lucky guessing would produce: 1 means perfect agreement, 0 means no better than chance.
- annotationannotation, n.: Adding information to raw data by hand or with software — marking objects in photos, tagging names in sentences, transcribing speech or ranking chatbot answers — so machines can learn from it.
- label noiselabel noise, n.: Mistakes in the answers attached to training or test examples — a dog photo marked 'cat' — which can mislead what a model learns and distort how good it seems.
- ground truthground truth, n.: The reference answers treated as correct when checking or training an AI system — for example, the confirmed diagnoses that a medical-imaging model's guesses are compared against.
- data labelingdata labeling, n.: The work of attaching correct answers to examples — often done by large paid workforces, sometimes by experts — so that AI systems can learn from them or be tested against them.
- human evaluationhuman evaluation, n.: Judging an AI's outputs by asking people to rate or compare them, because for many tasks — writing, conversation, translation — no automatic score captures quality well.
Sources
- Carletta, J. (1996). Assessing Agreement on Classification Tasks: The Kappa Statistic. Computational Linguistics 22(2).
- Artstein, R. & Poesio, M. (2008). Inter-Coder Agreement for Computational Linguistics. Computational Linguistics 34(4).
Mentioned in
annotationannotation, n.: Adding information to raw data by hand or with software — marking objects in photos, tagging names in sentences, transcribing speech or ranking chatbot answers — so machines can learn from it., crowdsourcingcrowdsourcing, n.: Getting many people over the internet to each do a small piece of a big job, such as labeling pictures for a few cents apiece, with anyone free to join in., data labelingdata labeling, n.: The work of attaching correct answers to examples — often done by large paid workforces, sometimes by experts — so that AI systems can learn from them or be tested against them., human evaluationhuman evaluation, n.: Judging an AI's outputs by asking people to rate or compare them, because for many tasks — writing, conversation, translation — no automatic score captures quality well., preference datapreference data, n.: Records of which of two or more AI answers people (or other AIs) liked better, collected so that a model can learn what good answers look like., Cohen's kappaCohen's kappa, n.: A score for how much two people labeling the same items agree beyond what lucky guessing would produce: 1 means perfect agreement, 0 means no better than chance., label noiselabel noise, n.: Mistakes in the answers attached to training or test examples — a dog photo marked 'cat' — which can mislead what a model learns and distort how good it seems.