language identification
In plain wordsAutomatically working out which language a piece of text or speech is in — a basic sorting step when building multilingual AI training data from the web.
The task of determining the language, and sometimes the script or dialect, of a text or audio segment. Text classifiers based on character n-gram statistics — fastText's language-ID models, Google's CLD family, langdetect — are fast enough to label billions of web documents and form a core stage of every multilingual data pipelinedata pipeline, n.: The chain of automated steps that collects raw data, cleans and transforms it, and delivers it in a usable form — the factory line that turns messy inputs into training material.: C4C4, prop. n.: A large collection of English web text, cleaned up from a 2019 web archive snapshot by Google researchers, that became a standard ingredient for training language models. kept pages its detector judged English with probability of at least 0.99, and FineWebFineWeb, prop. n.: A huge, free collection of cleaned-up English web text — around 15 trillion word fragments — released by Hugging Face in 2024 with a detailed public account of how it was built. kept documents scoring at least 0.65 for English. Accuracy is high for major languages and long texts but falls for short or mixed-language passages, closely related languages and the long tail of low-resource languageslow-resource language, n.: A language with little digital text, recorded speech or language tools available — often one spoken by millions — which makes AI systems much weaker in it than in English., where classifiers mistake noise, boilerplate or neighboring languages for the target. Audits of web-crawled multilingual corpora found that for many small languages most of the data labeled as that language was not usable text in it, prompting broader-coverage classifiers and per-language thresholds.
Language ID tagged thousands of pages of romanized Hindi as English, so we added a script check.
Origin
Automatic identification of a text's language predates the web; character n-gram statistics, as in William Cavnar and John Trenkle's 1994 text-categorization method, became an early standard. Isaac Caswell and colleagues (2020) and Julia Kreutzer and colleagues (2022) showed how badly it fails on web-scale data for small languages, shaping later multilingual pipelines.
Usage note
Spoken language identification is a separate speech task, often run before speech recognitionspeech recognition, n.: Technology that turns spoken words into written text, as when a phone transcribes a voice message or captions appear automatically under a video.. Errors in language ID silently shrink or pollute the data for exactly the languages that have least of it.
Related
- low-resource languagelow-resource language, n.: A language with little digital text, recorded speech or language tools available — often one spoken by millions — which makes AI systems much weaker in it than in English.
- data pipelinedata pipeline, n.: The chain of automated steps that collects raw data, cleans and transforms it, and delivers it in a usable form — the factory line that turns messy inputs into training material.
- quality filteringquality filtering, n.: Automatically sorting through huge piles of collected text and keeping only the parts judged useful for training, such as well-written articles, while discarding spam, gibberish and junk.
- FineWebFineWeb, prop. n.: A huge, free collection of cleaned-up English web text — around 15 trillion word fragments — released by Hugging Face in 2024 with a detailed public account of how it was built.
- C4C4, prop. n.: A large collection of English web text, cleaned up from a 2019 web archive snapshot by Google researchers, that became a standard ingredient for training language models.
- text classificationtext classification, n.: Sorting pieces of text into categories automatically — what topic an article covers, whether a message is spam, which team should handle a customer's complaint.
Sources
- Cavnar, W. B. & Trenkle, J. M. (1994). N-Gram-Based Text Categorization. Proceedings of SDAIR-94.
- Caswell, I., Breiner, T., van Esch, D. & Bapna, A. (2020). Language ID in the Wild: Unexpected Challenges on the Path to a Thousand-Language Web Text Corpus. COLING 2020.
Mentioned in
low-resource languagelow-resource language, n.: A language with little digital text, recorded speech or language tools available — often one spoken by millions — which makes AI systems much weaker in it than in English.