tokenizer
In plain wordsThe component that splits text into tokens and converts them to numbers before an AI model sees it, then turns the model's output numbers back into text.
The software component, together with its learned vocabularyvocabulary, n.: The fixed list of all the tokens a language model knows, each with its own number. Everything the model reads or writes must be built from pieces on this list. and rules, that performs tokenizationtokenization, n.: The first step in feeding text to a language model: chopping it into tokens — whole words, word pieces or characters — and turning each into a number the model can work with.: it normalizes input text (for example Unicode normalization), pre-splits it, often with a regular expression that separates words, punctuation and digits, applies a subword algorithm such as byte-pair encodingbyte-pair encoding, n.: A popular way to choose the word pieces an AI breaks text into: start from single characters and repeatedly glue together the pair that occurs most often, until the list reaches the desired size. merges, WordPieceWordPiece, n.: Google's method for splitting words into smaller known pieces, used by BERT. An unfamiliar word such as 'unhappiness' might be broken into parts like 'un', '##happi' and '##ness'. or unigram segmentation, inserts special tokensspecial token, n.: A reserved symbol in an AI model's vocabulary that is not ordinary text but a signal — 'start of message', 'end of answer', 'begin thinking' — used to structure what the model reads and writes., and returns integer IDs; the inverse maps IDs back to text. A tokenizer is trained on a corpus separately from, and before, the model, and is then frozen: the model's embeddingembedding, n.: A list of numbers that stands for a word, picture or other item, arranged so that similar items get similar lists. It lets a computer measure similarity of meaning as distance. and unembeddingunembedding, n.: The last step of a language model, which turns its internal state into a score for every token in its vocabulary; the next token is then chosen from those scores. matrices are indexed by its vocabulary, so changing it normally requires retraining or careful adaptation. Widely used implementations include SentencePiece, OpenAI's tiktoken and Hugging Face's tokenizers library. Vocabulary sizes grew from about 30,000 (BERT) and 50,000 (GPT-2) to roughly 100,000–260,000 in many models released in 2024–2026.
The new tokenizer packs Hindi text into far fewer tokens, cutting both cost and latency.
Origin
'Tokenizer' for a program that splits text into tokens comes from compiler construction and early NLP tooling. Trained subword tokenizers became standard, separately distributed components of neural models after Taku Kudo and John Richardson's SentencePiece (2018) and the byte-level BPE tokenizer of OpenAI's GPT-2 (2019).
Usage note
Mismatches between training and inference — wrong tokenizer version, missing special tokens, different whitespace handling — silently degrade outputs and are a common source of bugs.
Related
- tokenizationtokenization, n.: The first step in feeding text to a language model: chopping it into tokens — whole words, word pieces or characters — and turning each into a number the model can work with.
- tokentoken, n.: The small chunk of text an AI language model reads and writes — often a word, part of a word or a punctuation mark. AI usage, limits and prices are counted in tokens.
- vocabularyvocabulary, n.: The fixed list of all the tokens a language model knows, each with its own number. Everything the model reads or writes must be built from pieces on this list.
- byte-pair encodingbyte-pair encoding, n.: A popular way to choose the word pieces an AI breaks text into: start from single characters and repeatedly glue together the pair that occurs most often, until the list reaches the desired size.
- SentencePieceSentencePiece, prop. n.: Free software from Google that learns how to split text into tokens straight from raw sentences in any language, treating spaces as ordinary characters, so no language-specific word splitting is needed.
- special tokenspecial token, n.: A reserved symbol in an AI model's vocabulary that is not ordinary text but a signal — 'start of message', 'end of answer', 'begin thinking' — used to structure what the model reads and writes.
- chat templatechat template, n.: The exact text layout a chat AI was trained on for marking who is speaking — system, user or assistant. Software must reproduce it precisely, or the model performs noticeably worse.
Mentioned in
tokentoken, n.: The small chunk of text an AI language model reads and writes — often a word, part of a word or a punctuation mark. AI usage, limits and prices are counted in tokens., byte-pair encodingbyte-pair encoding, n.: A popular way to choose the word pieces an AI breaks text into: start from single characters and repeatedly glue together the pair that occurs most often, until the list reaches the desired size., compute-optimal trainingcompute-optimal training, n.: Choosing a model's size and the amount of data it trains on so that a fixed amount of computing produces the best possible model — neither too big and undertrained nor too small and overfed., low-resource languagelow-resource language, n.: A language with little digital text, recorded speech or language tools available — often one spoken by millions — which makes AI systems much weaker in it than in English., perplexityperplexity, n.: A score for how surprised a language model is by real text; lower is better. A perplexity of 10 means guessing each next word is, on average, as hard for the model as guessing a ten-sided die's roll., special tokenspecial token, n.: A reserved symbol in an AI model's vocabulary that is not ordinary text but a signal — 'start of message', 'end of answer', 'begin thinking' — used to structure what the model reads and writes., tokenizationtokenization, n.: The first step in feeding text to a language model: chopping it into tokens — whole words, word pieces or characters — and turning each into a number the model can work with., tokens per parametertokens per parameter, n.: How many pieces of text a language model reads during training for each adjustable number inside it — a quick way to tell whether a model was trained on a little or a lot of data for its size., tokens per secondtokens per second, n.: A speed rating for AI text generation: how many word-pieces a model produces each second. Five to ten keeps up with a fast reader; hundreds feel instant., vocabularyvocabulary, n.: The fixed list of all the tokens a language model knows, each with its own number. Everything the model reads or writes must be built from pieces on this list., byte-level modelbyte-level model, n.: A language model that reads raw bytes — the basic units computers use to store text — instead of tokens from a fixed list, avoiding a separate tokenizer but making inputs much longer., chat templatechat template, n.: The exact text layout a chat AI was trained on for marking who is speaking — system, user or assistant. Software must reproduce it precisely, or the model performs noticeably worse., GGUFGGUF, abbr.: A file format that packs an AI language model, often in a shrunken low-precision version, into one file that ordinary computers can load and run quickly; used by llama.cpp and Ollama., glitch tokenglitch token, n.: A word fragment in an AI model's vocabulary that the model almost never saw during training, so asking about it can produce bizarre, evasive or garbled answers., image tokenizerimage tokenizer, n.: A component that chops a picture into a short sequence of 'visual words' that a language-style AI model can read or write, and turns such sequences back into pictures., irreducible lossirreducible loss, n.: The part of a model's error that no amount of extra size, data or training can remove, because the thing being predicted is itself partly random — like the unpredictability of the next word in an ordinary sentence., logit biaslogit bias, n.: A setting that lets developers make specific words more or less likely in an AI model's output — even ban a word outright — by adjusting the model's scores before it chooses., SentencePieceSentencePiece, prop. n.: Free software from Google that learns how to split text into tokens straight from raw sentences in any language, treating spaces as ordinary characters, so no language-specific word splitting is needed., TransformersTransformers, prop. n.: A free software library from Hugging Face that lets programmers download and use thousands of ready-trained AI models for text, images and sound with a few lines of code., WordPieceWordPiece, n.: Google's method for splitting words into smaller known pieces, used by BERT. An unfamiliar word such as 'unhappiness' might be broken into parts like 'un', '##happi' and '##ness'.