low-resource language
In plain wordsA language with little digital text, recorded speech or language tools available — often one spoken by millions — which makes AI systems much weaker in it than in English.
A language for which little machine-readable data (raw text, transcribed speech, parallel translations, labeled datasets) and few tools (spell-checkers, tokenizers, benchmarks) exist, relative to its number of speakers or its importance. Most of the world's roughly 7,000 languages are low-resource, including some with tens of millions of speakers, while a handful led by English dominate web text. Consequences for AI include weaker comprehension and generation, more hallucinationshallucination, n.: When an AI states something false or made up — a fake fact, quotation or source — fluently and confidently, as if it were true. and safety failures, and higher costs, because tokenizerstokenizer, n.: The component that splits text into tokens and converts them to numbers before an AI model sees it, then turns the model's output numbers back into text. trained mainly on English split other languages into many more tokens. Web-crawled data labeled as belonging to such languages is often mislabeled, boilerplate or machine-translated. Remedies include multilingual pretraining with transfer from related languages, per-language language identificationlanguage identification, n.: Automatically working out which language a piece of text or speech is in — a basic sorting step when building multilingual AI training data from the web. and filtering, community-led data collection and dedicated benchmarks; initiatives such as Masakhane, Meta's No Language Left Behind, Cohere's Aya and FineWeb2 target the gap.
Our model writes fluent Swahili but stumbles in Tigrinya, a low-resource language with little web text.
Origin
In use in computational linguistics and speech technology by the 2000s, alongside 'less-resourced', 'under-resourced' and 'low-density'. Pratik Joshi and colleagues' 2020 taxonomy sorted the world's languages into six classes by available labeled and unlabeled data and found that most fall in the poorest classes.
Usage note
'Low-resource' describes data availability, not a language's richness or its number of speakers, and some speakers object to the deficit framing. A language can be well resourced in text but not speech, or the reverse.
Related
- language identificationlanguage identification, n.: Automatically working out which language a piece of text or speech is in — a basic sorting step when building multilingual AI training data from the web.
- tokenizationtokenization, n.: The first step in feeding text to a language model: chopping it into tokens — whole words, word pieces or characters — and turning each into a number the model can work with.
- machine translationmachine translation, n.: Software that renders writing or speech in one language into another language, as online translation services and phone translation apps do.
- natural language processingnatural language processing, n.: The field that teaches computers to handle human language — to read, interpret, translate, summarize and produce text and speech. It is one of the oldest and largest areas of artificial intelligence.
- sampling biassampling bias, n.: A distortion that arises when the data collected over-represents some kinds of people or situations and under-represents others, so conclusions drawn from it don't hold for everyone.
- digital dividedigital divide, n.: The gap between people, places or countries that have good access to digital technology, and the skills to use it, and those that do not; AI may widen it.
- tokenizertokenizer, n.: The component that splits text into tokens and converts them to numbers before an AI model sees it, then turns the model's output numbers back into text.
- transfer learningtransfer learning, n.: Reusing what a model learned on one task to get a head start on another — like a pianist picking up the organ faster than a beginner. It lets models learn new jobs from little data.
- pretraining datapretraining data, n.: The enormous body of text, code and other material — mostly from the web — that a large AI model first learns from, before it is tuned to follow instructions or chat.
- AI colonialismAI colonialism, n.: The criticism that the AI industry repeats patterns of colonial empires, taking data, labor, land and resources from poorer places and peoples while profits and control stay with a few rich companies and countries.
Sources
- Joshi, P., Santy, S., Budhiraja, A., Bali, K. & Choudhury, M. (2020). The State and Fate of Linguistic Diversity and Inclusion in the NLP World. ACL 2020.
- Petrov, A., La Malfa, E., Torr, P. H. S. & Bibi, A. (2023). Language Model Tokenizers Introduce Unfairness Between Languages. NeurIPS 2023.
- Kreutzer, J. et al. (2022). Quality at a Glance: An Audit of Web-Crawled Multilingual Datasets. Transactions of the ACL 10.
Mentioned in
machine translationmachine translation, n.: Software that renders writing or speech in one language into another language, as online translation services and phone translation apps do., language identificationlanguage identification, n.: Automatically working out which language a piece of text or speech is in — a basic sorting step when building multilingual AI training data from the web., part-of-speech taggingpart-of-speech tagging, n.: Labeling each word in a sentence with its grammatical role — noun, verb, adjective and so on — as in 'Time (noun) flies (verb) like (preposition) an arrow'.