part-of-speech tagging
In plain wordsLabeling each word in a sentence with its grammatical role — noun, verb, adjective and so on — as in 'Time (noun) flies (verb) like (preposition) an arrow'.
The task of assigning each word (token) in a text a grammatical category from a tagset, such as noun, verb, adjective or determiner, often with finer distinctions such as plural noun or past-tense verb. The difficulty is ambiguity — 'book' may be a noun or a verb, 'that' a determiner, pronoun or conjunction — resolved from context. Classic approaches were rule-based taggers, hidden Markov modelshidden Markov model, n.: A model of a sequence, such as speech or DNA, in which hidden states — say, the words being spoken — change step by step at random, and each state gives off signals that can be observed, such as sounds. decoded with the Viterbi algorithm, transformation-based learning, and maximum-entropy and conditional-random-field models; neural taggers followed. English taggers trained on newswire exceed 97% per-token accuracy, close to agreement between human annotators. Once a standard first stage of natural language processingnatural language processing, n.: The field that teaches computers to handle human language — to read, interpret, translate, summarize and produce text and speech. It is one of the oldest and largest areas of artificial intelligence. pipelines, feeding syntactic parsingsyntactic parsing, n.: Working out the grammatical structure of a sentence — which words group together and which word modifies which — usually drawn as a tree. and information extractioninformation extraction, n.: Automatically pulling structured facts out of ordinary text — who did what, when, where and for how much — and putting them into a table or database., it is now mostly implicit in neural models, though still used in linguistics, corpus annotation and work on low-resource languageslow-resource language, n.: A language with little digital text, recorded speech or language tools available — often one spoken by millions — which makes AI systems much weaker in it than in English..
The tagger labeled 'flies' as a noun, so the parse fell apart.
Origin
Automatic tagging began with the Brown CorpusBrown Corpus, prop. n.: The first large collection of English text stored on computer for research: about a million words from 500 American publications of 1961, assembled at Brown University in the 1960s. of American English (1960s), much of which was tagged with the rule-based TAGGIT program in the early 1970s. Statistical taggers by Kenneth Church and Steven DeRose (1988) and the Penn Treebank tagset (Mitchell Marcus and colleagues, 1993) defined the field's standard setting.
Related
- syntactic parsingsyntactic parsing, n.: Working out the grammatical structure of a sentence — which words group together and which word modifies which — usually drawn as a tree.
- natural language processingnatural language processing, n.: The field that teaches computers to handle human language — to read, interpret, translate, summarize and produce text and speech. It is one of the oldest and largest areas of artificial intelligence.
- hidden Markov modelhidden Markov model, n.: A model of a sequence, such as speech or DNA, in which hidden states — say, the words being spoken — change step by step at random, and each state gives off signals that can be observed, such as sounds.
- corpuscorpus, n.: A large collection of written or spoken language, gathered so that people or computers can study it or learn from it — like a library assembled for a language model.
- named-entity recognitionnamed-entity recognition, n.: Software that spots names and similar expressions in text and labels each by type: in 'Maria flew from Lagos to Paris on Monday', it marks a person, two cities and a date.
- computational linguisticscomputational linguistics, n.: The science of human language studied with computers: using computational models to understand how language works, and building the language technology that grows out of that understanding.
- Brown CorpusBrown Corpus, prop. n.: The first large collection of English text stored on computer for research: about a million words from 500 American publications of 1961, assembled at Brown University in the 1960s.
Sources
- Marcus, M. P., Santorini, B. & Marcinkiewicz, M. A. (1993). Building a Large Annotated Corpus of English: The Penn Treebank. Computational Linguistics 19(2), 313–330.
- Church, K. W. (1988). A Stochastic Parts Program and Noun Phrase Parser for Unrestricted Text. Second Conference on Applied Natural Language Processing (ANLP 1988).
Mentioned in
natural language processingnatural language processing, n.: The field that teaches computers to handle human language — to read, interpret, translate, summarize and produce text and speech. It is one of the oldest and largest areas of artificial intelligence., computational linguisticscomputational linguistics, n.: The science of human language studied with computers: using computational models to understand how language works, and building the language technology that grows out of that understanding., conditional random fieldconditional random field, n.: A statistical method for labeling every item in a sequence, such as marking each word in a sentence as a name, a place or an ordinary word, by choosing all the labels together rather than one at a time., hidden Markov modelhidden Markov model, n.: A model of a sequence, such as speech or DNA, in which hidden states — say, the words being spoken — change step by step at random, and each state gives off signals that can be observed, such as sounds., named-entity recognitionnamed-entity recognition, n.: Software that spots names and similar expressions in text and labels each by type: in 'Maria flew from Lagos to Paris on Monday', it marks a person, two cities and a date., Penn TreebankPenn Treebank, prop. n.: A collection of English text, best known for its million words of newspaper articles, in which experts marked each word's grammatical role and drew each sentence's structure as a tree; it trained and tested a generation of language software., Viterbi algorithmViterbi algorithm, n.: A method that finds the likeliest hidden sequence behind some observations — such as each word's grammatical role in a sentence — by keeping only the best path to each point instead of trying every combination., stemmingstemming, n.: Cutting words down to a common root so that variants count as the same word — 'connected', 'connecting' and 'connection' all become 'connect' — to help search and text analysis., syntactic parsingsyntactic parsing, n.: Working out the grammatical structure of a sentence — which words group together and which word modifies which — usually drawn as a tree.