special token
In plain wordsA reserved symbol in an AI model's vocabulary that is not ordinary text but a signal — 'start of message', 'end of answer', 'begin thinking' — used to structure what the model reads and writes.
A tokentoken, n.: The small chunk of text an AI language model reads and writes — often a word, part of a word or a punctuation mark. AI usage, limits and prices are counted in tokens. reserved in a model's vocabularyvocabulary, n.: The fixed list of all the tokens a language model knows, each with its own number. Everything the model reads or writes must be built from pieces on this list. for structural or control purposes rather than to represent text, and normally never produced when ordinary input is tokenized. Examples include beginning- and end-of-sequence markers (<s>, </s>, <|endoftext|>), padding and unknown-word tokens, BERTBERT, prop. n.: An influential 2018 Google language model that learned to understand text by filling in hidden words. It made search engines and many language tools much better at grasping meaning in context.'s [CLS], [SEP] and [MASK], chat templatechat template, n.: The exact text layout a chat AI was trained on for marking who is speaking — system, user or assistant. Software must reproduce it precisely, or the model performs noticeably worse. markers for roles and turn ends, delimiters for reasoningreasoning trace, n.: The written-out 'thinking' an AI reasoning model produces before its final answer — notes to itself full of attempts, checks and corrections, which users may or may not get to see. and tool callstool call, n.: A single request from an AI model to use one of its tools — naming the tool and filling in the details — which the surrounding software then carries out and reports back on., and placeholders for images or audio in multimodal models. The model learns their meaning in training; emitting an end-of-turn token is how a chat model signals that it has finished. Tokenizers should refuse to parse such strings in untrusted text, since letting users inject them can spoof roles and undermine the instruction hierarchyinstruction hierarchy, n.: A rule built into AI assistants about whose instructions win when they conflict: the developer's beat the user's, and the user's beat any instructions hidden in a web page or document..
Generation never stopped because the end-of-turn special token wasn't in the stop list.
Origin
Reserved start and end symbols date from early neural sequence-to-sequencesequence-to-sequence model, n.: A model that turns one sequence into another, such as a French sentence into English or speech into text, by first reading the whole input and then writing the output step by step. work: Ilya Sutskever and colleagues (2014) appended an end-of-sentence symbol to every sentence. BERTBERT, prop. n.: An influential 2018 Google language model that learned to understand text by filling in hidden words. It made search engines and many language tools much better at grasping meaning in context. (2018) popularized named special tokens such as [CLS] and [MASK], and chat and reasoning models from 2022 multiplied them.
Related
- tokentoken, n.: The small chunk of text an AI language model reads and writes — often a word, part of a word or a punctuation mark. AI usage, limits and prices are counted in tokens.
- tokenizertokenizer, n.: The component that splits text into tokens and converts them to numbers before an AI model sees it, then turns the model's output numbers back into text.
- vocabularyvocabulary, n.: The fixed list of all the tokens a language model knows, each with its own number. Everything the model reads or writes must be built from pieces on this list.
- chat templatechat template, n.: The exact text layout a chat AI was trained on for marking who is speaking — system, user or assistant. Software must reproduce it precisely, or the model performs noticeably worse.
- stop sequencestop sequence, n.: A piece of text that tells an AI model to stop writing as soon as it produces it — for example, stopping at a blank line or just before the next question.
- glitch tokenglitch token, n.: A word fragment in an AI model's vocabulary that the model almost never saw during training, so asking about it can produce bizarre, evasive or garbled answers.
- BERTBERT, prop. n.: An influential 2018 Google language model that learned to understand text by filling in hidden words. It made search engines and many language tools much better at grasping meaning in context.
Sources
- Devlin, J., Chang, M.-W., Lee, K. & Toutanova, K. (2019). BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. NAACL 2019.
- Sutskever, I., Vinyals, O. & Le, Q. V. (2014). Sequence to Sequence Learning with Neural Networks. NeurIPS 2014.
Mentioned in
tokentoken, n.: The small chunk of text an AI language model reads and writes — often a word, part of a word or a punctuation mark. AI usage, limits and prices are counted in tokens., chat modelchat model, n.: A language model trained to hold a back-and-forth conversation, keeping track of who said what and replying as a helpful assistant. It is the engine inside chat apps., reasoning tracereasoning trace, n.: The written-out 'thinking' an AI reasoning model produces before its final answer — notes to itself full of attempts, checks and corrections, which users may or may not get to see., stop sequencestop sequence, n.: A piece of text that tells an AI model to stop writing as soon as it produces it — for example, stopping at a blank line or just before the next question., tokenizertokenizer, n.: The component that splits text into tokens and converts them to numbers before an AI model sees it, then turns the model's output numbers back into text., vocabularyvocabulary, n.: The fixed list of all the tokens a language model knows, each with its own number. Everything the model reads or writes must be built from pieces on this list., chat templatechat template, n.: The exact text layout a chat AI was trained on for marking who is speaking — system, user or assistant. Software must reproduce it precisely, or the model performs noticeably worse., fill-in-the-middlefill-in-the-middle, n.: A training trick that teaches a text-writing AI to fill a gap using what comes both before and after it — essential for coding assistants that insert code in the middle of a file.