tokenization
In plain wordsThe first step in feeding text to a language model: chopping it into tokens — whole words, word pieces or characters — and turning each into a number the model can work with.
The process of segmenting text into tokenstoken, n.: The small chunk of text an AI language model reads and writes — often a word, part of a word or a punctuation mark. AI usage, limits and prices are counted in tokens. and mapping each to an integer ID from a fixed vocabularyvocabulary, n.: The fixed list of all the tokens a language model knows, each with its own number. Everything the model reads or writes must be built from pieces on this list., performed by a tokenizertokenizer, n.: The component that splits text into tokens and converts them to numbers before an AI model sees it, then turns the model's output numbers back into text. before text enters a model and reversed (detokenization) on output. Classical NLP split text into words with whitespace and punctuation rules; modern language modelslarge language model, n.: A computer program trained on enormous amounts of text to predict the words that come next. At vast scale, that simple skill lets it chat, answer questions, translate, summarize and write code. use subword tokenization, typically byte-pair encodingbyte-pair encoding, n.: A popular way to choose the word pieces an AI breaks text into: start from single characters and repeatedly glue together the pair that occurs most often, until the list reaches the desired size., WordPieceWordPiece, n.: Google's method for splitting words into smaller known pieces, used by BERT. An unfamiliar word such as 'unhappiness' might be broken into parts like 'un', '##happi' and '##ness'. or the unigram method, so frequent words are single tokens, rare words break into pieces, and out-of-vocabulary input becomes rare (impossible with byte-level schemes). Tokenization has wide consequences: it sets how much text fits in a context windowcontext window, n.: How much an AI model can read and keep in view at once: the instructions, the conversation so far and any attached documents. Text that does not fit has to be cut, summarized or left out. and what a request costs, how fairly languages are served (some scripts need several times more tokens for the same content), and it underlies failures such as miscounting letters or mishandling digits, because the model never sees characters directly. Byte-level modelsbyte-level model, n.: A language model that reads raw bytes — the basic units computers use to store text — instead of tokens from a fixed list, avoiding a separate tokenizer but making inputs much longer. try to eliminate the step.
Half the arithmetic errors traced back to tokenization splitting long numbers into uneven chunks.
Origin
Rule-based word tokenization is as old as computational linguistics; the Penn Treebank conventions of the early 1990s became a de facto standard for English. Subword tokenization for neural models dates from 2015–2016, with Rico Sennrich, Barry Haddow and Alexandra Birch's byte-pair encoding and the wordpieces of Google's neural translation system.
Usage note
Many oddities of LLMs, such as the strawberry problemstrawberry problem, n.: The once-famous failure of AI chatbots to count the r's in 'strawberry' correctly — a simple question that exposed how these models see words in chunks rather than letters., are blamed on tokenization. Not to be confused with tokenization in data security (replacing sensitive values with surrogates) or in finance (representing assets as blockchain tokens).
Related
- tokentoken, n.: The small chunk of text an AI language model reads and writes — often a word, part of a word or a punctuation mark. AI usage, limits and prices are counted in tokens.
- tokenizertokenizer, n.: The component that splits text into tokens and converts them to numbers before an AI model sees it, then turns the model's output numbers back into text.
- byte-pair encodingbyte-pair encoding, n.: A popular way to choose the word pieces an AI breaks text into: start from single characters and repeatedly glue together the pair that occurs most often, until the list reaches the desired size.
- WordPieceWordPiece, n.: Google's method for splitting words into smaller known pieces, used by BERT. An unfamiliar word such as 'unhappiness' might be broken into parts like 'un', '##happi' and '##ness'.
- SentencePieceSentencePiece, prop. n.: Free software from Google that learns how to split text into tokens straight from raw sentences in any language, treating spaces as ordinary characters, so no language-specific word splitting is needed.
- vocabularyvocabulary, n.: The fixed list of all the tokens a language model knows, each with its own number. Everything the model reads or writes must be built from pieces on this list.
- byte-level modelbyte-level model, n.: A language model that reads raw bytes — the basic units computers use to store text — instead of tokens from a fixed list, avoiding a separate tokenizer but making inputs much longer.
- strawberry problemstrawberry problem, n.: The once-famous failure of AI chatbots to count the r's in 'strawberry' correctly — a simple question that exposed how these models see words in chunks rather than letters.
Sources
- Sennrich, R., Haddow, B. & Birch, A. (2016). Neural Machine Translation of Rare Words with Subword Units. ACL 2016. arxiv.org/abs/1508.07909
Mentioned in
tokentoken, n.: The small chunk of text an AI language model reads and writes — often a word, part of a word or a punctuation mark. AI usage, limits and prices are counted in tokens., byte-pair encodingbyte-pair encoding, n.: A popular way to choose the word pieces an AI breaks text into: start from single characters and repeatedly glue together the pair that occurs most often, until the list reaches the desired size., low-resource languagelow-resource language, n.: A language with little digital text, recorded speech or language tools available — often one spoken by millions — which makes AI systems much weaker in it than in English., strawberry problemstrawberry problem, n.: The once-famous failure of AI chatbots to count the r's in 'strawberry' correctly — a simple question that exposed how these models see words in chunks rather than letters., tokenizertokenizer, n.: The component that splits text into tokens and converts them to numbers before an AI model sees it, then turns the model's output numbers back into text., vocabularyvocabulary, n.: The fixed list of all the tokens a language model knows, each with its own number. Everything the model reads or writes must be built from pieces on this list., byte-level modelbyte-level model, n.: A language model that reads raw bytes — the basic units computers use to store text — instead of tokens from a fixed list, avoiding a separate tokenizer but making inputs much longer., seahorse emojiseahorse emoji, n.: A viral meme in which chatbots, asked whether a seahorse emoji exists, insisted it did and then spiraled through wrong emojis trying to produce one — though there was no such emoji., SentencePieceSentencePiece, prop. n.: Free software from Google that learns how to split text into tokens straight from raw sentences in any language, treating spaces as ordinary characters, so no language-specific word splitting is needed., stemmingstemming, n.: Cutting words down to a common root so that variants count as the same word — 'connected', 'connecting' and 'connection' all become 'connect' — to help search and text analysis., WordPieceWordPiece, n.: Google's method for splitting words into smaller known pieces, used by BERT. An unfamiliar word such as 'unhappiness' might be broken into parts like 'un', '##happi' and '##ness'., Zipf's lawZipf's law, n.: The pattern that word use is lopsided: in a big text the top word is roughly twice as common as the runner-up and ten times as common as the tenth, so a few words dominate and most are rare.