annotation
In plain wordsAdding information to raw data by hand or with software — marking objects in photos, tagging names in sentences, transcribing speech or ranking chatbot answers — so machines can learn from it.
The process, and the resulting layer of information, by which people or programs add structured descriptions to raw data so that it can be used for training or evaluation: class labelslabel, n.: The correct answer attached to an example in a dataset — 'cat' on a photo of a cat, 'positive' on an upbeat review — that a learning system is trained to predict., bounding boxesbounding box, n.: The rectangle a vision system draws around an object it has found, stored as a handful of numbers giving the rectangle's position and size. and segmentation masks in images; part-of-speech tags, named entitiesnamed-entity recognition, n.: Software that spots names and similar expressions in text and labels each by type: in 'Maria flew from Lagos to Paris on Monday', it marks a person, two cities and a date. and parse trees in text; timestamps and transcripts in audio; and rankings or critiques of model outputs for reinforcement learning from human feedbackreinforcement learning from human feedback, n.: A way of training AI using people's judgments: humans compare pairs of answers, a scoring model learns their preferences, and the AI is then trained to produce answers that score well.. Annotation projects depend on written guidelines that define categories and edge cases, pilot rounds, quality checks such as inter-annotator agreementinter-annotator agreement, n.: A measure of how often different people labeling the same data give the same answer. Low agreement warns that the task is unclear or the labels are unreliable., and adjudication of disagreements. Increasingly, models pre-annotate data for humans to correct, or serve as annotators themselves (see LLM-as-a-judgeLLM-as-a-judge, n.: Using one AI language model to grade or compare the answers of another, as a cheaper and faster stand-in for human reviewers.).
Two annotators disagreed on 18% of the sarcasm examples, so we rewrote the guidelines.
Origin
Annotating texts is centuries old, but the machine-readable sense comes from corpus linguistics: part-of-speech tagging of the Brown Corpus in the 1970s, then the Penn Treebank (Mitchell Marcus and colleagues, 1993), whose syntactic annotation of Wall Street Journal text trained a generation of statistical parsers.
Usage note
Often used interchangeably with data labelingdata labeling, n.: The work of attaching correct answers to examples — often done by large paid workforces, sometimes by experts — so that AI systems can learn from them or be tested against them.. Where a distinction is drawn, annotation is the broader term (any added structure, including spans, relations and free-text rationales) and labeling is the assignment of target values, often as an outsourced industrial service.
Related
- labellabel, n.: The correct answer attached to an example in a dataset — 'cat' on a photo of a cat, 'positive' on an upbeat review — that a learning system is trained to predict.
- data labelingdata labeling, n.: The work of attaching correct answers to examples — often done by large paid workforces, sometimes by experts — so that AI systems can learn from them or be tested against them.
- inter-annotator agreementinter-annotator agreement, n.: A measure of how often different people labeling the same data give the same answer. Low agreement warns that the task is unclear or the labels are unreliable.
- crowdsourcingcrowdsourcing, n.: Getting many people over the internet to each do a small piece of a big job, such as labeling pictures for a few cents apiece, with anyone free to join in.
- ground truthground truth, n.: The reference answers treated as correct when checking or training an AI system — for example, the confirmed diagnoses that a medical-imaging model's guesses are compared against.
- golden datasetgolden dataset, n.: A small, carefully checked set of example questions with known good answers, used as a trusted yardstick to test an AI system every time it changes.
- Brown CorpusBrown Corpus, prop. n.: The first large collection of English text stored on computer for research: about a million words from 500 American publications of 1961, assembled at Brown University in the 1960s.
Sources
- Marcus, M. P., Santorini, B. & Marcinkiewicz, M. A. (1993). Building a Large Annotated Corpus of English: The Penn Treebank. Computational Linguistics 19(2).
Mentioned in
datasetdataset, n.: An organized collection of examples gathered for a purpose — such as photos with their captions, or questions with answers — used to teach or test an AI system., labellabel, n.: The correct answer attached to an example in a dataset — 'cat' on a photo of a cat, 'positive' on an upbeat review — that a learning system is trained to predict., active learningactive learning, n.: A way to train a model with fewer labeled examples by letting it choose which unlabeled examples a person should label next, usually the ones it is most unsure about., bounding boxbounding box, n.: The rectangle a vision system draws around an object it has found, stored as a handful of numbers giving the rectangle's position and size., corpuscorpus, n.: A large collection of written or spoken language, gathered so that people or computers can study it or learn from it — like a library assembled for a language model., crowdsourcingcrowdsourcing, n.: Getting many people over the internet to each do a small piece of a big job, such as labeling pictures for a few cents apiece, with anyone free to join in., data labelingdata labeling, n.: The work of attaching correct answers to examples — often done by large paid workforces, sometimes by experts — so that AI systems can learn from them or be tested against them., ghost workghost work, n.: The hidden human labor behind AI: people, often poorly paid and far away, who label data, check outputs and handle cases the software cannot, while the product appears fully automatic., ground truthground truth, n.: The reference answers treated as correct when checking or training an AI system — for example, the confirmed diagnoses that a medical-imaging model's guesses are compared against., Segment Anything ModelSegment Anything Model, prop. n.: A family of AI models from Meta that trace the exact outline of whatever you point to in a photo or video, whether by clicking on it, drawing a box around it or, in newer versions, naming it., Amazon Mechanical TurkAmazon Mechanical Turk, prop. n.: A website Amazon opened in 2005 that acts as a hiring hall for tiny online jobs, such as labeling images for AI, which registered workers pick up and finish for a few cents each., Brown CorpusBrown Corpus, prop. n.: The first large collection of English text stored on computer for research: about a million words from 500 American publications of 1961, assembled at Brown University in the 1960s., Cohen's kappaCohen's kappa, n.: A score for how much two people labeling the same items agree beyond what lucky guessing would produce: 1 means perfect agreement, 0 means no better than chance., inter-annotator agreementinter-annotator agreement, n.: A measure of how often different people labeling the same data give the same answer. Low agreement warns that the task is unclear or the labels are unreliable.