glitch token

A tokentoken, n.: The small chunk of text an AI language model reads and writes — often a word, part of a word or a punctuation mark. AI usage, limits and prices are counted in tokens. in a model's vocabularyvocabulary, n.: The fixed list of all the tokens a language model knows, each with its own number. Everything the model reads or writes must be built from pieces on this list. that is under-trained because it was frequent in the text used to build the tokenizertokenizer, n.: The component that splits text into tokens and converts them to numbers before an AI model sees it, then turns the model's output numbers back into text. but rare or absent in the model's own training data, leaving its embeddingembedding, n.: A list of numbers that stands for a word, picture or other item, arranged so that similar items get similar lists. It lets a computer measure similarity of meaning as distance. close to its initial random value. Prompts containing such tokens can make a model fail to repeat them, substitute other words, turn evasive or produce unrelated text. Typical sources are usernames, strings from scraped forums or logs, and fragments of long repeated sequences. The best-known example is ' SolidGoldMagikarp', a Reddit username that GPT-3-era models could not reproduce. Glitch tokens can be found systematically by inspecting embedding norms or unembeddingunembedding, n.: The last step of a language model, which turns its internal state into a score for every token in its vocabulary; the next token is then chosen from those scores. statistics and avoided by training tokenizers on data representative of model training; they also concern security researchers because they trigger unpredictable behavior.

The scan flagged 40 glitch tokens, mostly usernames scraped from a counting forum.