stemming
In plain wordsCutting words down to a common root so that variants count as the same word — 'connected', 'connecting' and 'connection' all become 'connect' — to help search and text analysis.
A text-normalization step that chops affixes, mainly suffixes, off words by heuristic rules, so that different forms of a word reduce to one shared key, the stem — 'running' and 'runs' both become 'run' — and can match in search or be counted together. The key can be a non-word ('studies' becomes 'studi'), and errors run both ways: over-stemming merges unrelated words (an aggressive stemmer may reduce 'organization' and 'organ' to the same key), while under-stemming leaves true variants apart. Lemmatization is the more linguistically informed alternative, mapping each word to its dictionary form (lemma) using a vocabulary and part-of-speechpart-of-speech tagging, n.: Labeling each word in a sentence with its grammatical role — noun, verb, adjective and so on — as in 'Time (noun) flies (verb) like (preposition) an arrow'. information, so that 'better' becomes 'good'. Both were staples of information retrievalinformation retrieval, n.: The study and engineering of search: how to organize huge piles of documents so that the few that matter to someone's question can be found and ranked quickly. AI systems rely on it to look things up., bag of wordsbag of words, n.: A simple way of turning text into numbers by counting how often each word appears and ignoring word order — like describing a recipe only by its shopping list. features and topic modelstopic model, n.: A method that reads a large pile of documents and discovers the themes running through them, describing each theme by its typical words and each document by its mix of themes.; neural models built on subword tokenizationtokenization, n.: The first step in feeding text to a language model: chopping it into tokens — whole words, word pieces or characters — and turning each into a number the model can work with. rarely need them, but search engines and morphologically rich languages still use them.
Turn on stemming so a search for 'invoices' also finds 'invoiced' and 'invoicing'.
Origin
Julie Beth Lovins published the first widely cited stemming algorithm in 1968. Martin Porter's suffix-stripping algorithm (1980) became the standard for English, and his Snowball framework (early 2000s) extended the approach to many languages.
Related
- information retrievalinformation retrieval, n.: The study and engineering of search: how to organize huge piles of documents so that the few that matter to someone's question can be found and ranked quickly. AI systems rely on it to look things up.
- bag of wordsbag of words, n.: A simple way of turning text into numbers by counting how often each word appears and ignoring word order — like describing a recipe only by its shopping list.
- tokenizationtokenization, n.: The first step in feeding text to a language model: chopping it into tokens — whole words, word pieces or characters — and turning each into a number the model can work with.
- TF-IDFTF-IDF, abbr.: A simple score for picking out the words that characterize a document: a word counts for more the more often it appears there, and for less the more of the collection's other documents also use it.
- part-of-speech taggingpart-of-speech tagging, n.: Labeling each word in a sentence with its grammatical role — noun, verb, adjective and so on — as in 'Time (noun) flies (verb) like (preposition) an arrow'.
- sparse retrievalsparse retrieval, n.: Finding documents by the words they contain, giving more weight to rare, telling words — the classic way search engines work.
Sources
- Porter, M. F. (1980). An algorithm for suffix stripping. Program 14(3), 130–137.
- Lovins, J. B. (1968). Development of a Stemming Algorithm. Mechanical Translation and Computational Linguistics 11(1–2), 22–31.