tokens per parameter
In plain wordsHow many pieces of text a language model reads during training for each adjustable number inside it — a quick way to tell whether a model was trained on a little or a lot of data for its size.
The ratio D/N of training tokenstoken, n.: The small chunk of text an AI language model reads and writes — often a word, part of a word or a punctuation mark. AI usage, limits and prices are counted in tokens. to model parameters, a compact descriptor of where a training runtraining run, n.: One complete attempt at training a model with a fixed recipe, from start to finish. For the largest AI models a single run can occupy tens of thousands of chips for months. sits relative to compute-optimality. The Chinchilla scaling lawsChinchilla scaling laws, n.: A 2022 DeepMind finding that, for a fixed amount of computing, language models had been made too big and trained on too little text; the best balance is roughly 20 word-pieces of text for every adjustable number inside the model. put the training-compute optimum for dense transformers near 20. GPT-3GPT-3, prop. n.: A huge 2020 OpenAI language model that could perform new tasks simply from instructions or a few examples written into its prompt, without retraining. It made prompting a new way of programming. (175 billion parameters, 300 billion tokens) sat below 2, while models built for cheap inference sit far above it: Meta's Llama 3 8B, trained on over 15 trillion tokens, is near 1,900. The ratio counts tokens seen, including repeats, so it says nothing about data diversity; it depends on the tokenizertokenizer, n.: The component that splits text into tokens and converts them to numbers before an AI model sees it, then turns the model's output numbers back into text., since a more efficient one yields fewer tokens for the same text; and for mixture-of-expertsmixture of experts, n.: A way to build very large AI models cheaply: the model contains many specialist sub-networks, and for each word only a few are switched on, picked by a small dispatcher network. models it is usually computed with active rather than total parameters, a convention that varies.
At roughly 300 tokens per parameter, the 7B model is heavily overtrained by Chinchilla standards.
Origin
The ratio became standard shorthand after DeepMind's Chinchilla paper (Jordan Hoffmann et al., 2022) implied about 20 tokens per parameter; the '20 tokens per parameter' rule of thumb spread in 2022–2023 as open models were deliberately trained far beyond it.
Usage note
A high ratio is not a defect: it usually signals overtrainingovertraining, n.: Deliberately training a model on much more data than would be most efficient for its training budget, so that a smaller model reaches good quality and is cheaper to run afterwards. chosen to cut inference cost. Ratios computed with different tokenizers, or for sparse models, are not directly comparable.
Related
- Chinchilla scaling lawsChinchilla scaling laws, n.: A 2022 DeepMind finding that, for a fixed amount of computing, language models had been made too big and trained on too little text; the best balance is roughly 20 word-pieces of text for every adjustable number inside the model.
- compute-optimal trainingcompute-optimal training, n.: Choosing a model's size and the amount of data it trains on so that a fixed amount of computing produces the best possible model — neither too big and undertrained nor too small and overfed.
- overtrainingovertraining, n.: Deliberately training a model on much more data than would be most efficient for its training budget, so that a smaller model reaches good quality and is cheaper to run afterwards.
- parameter countparameter count, n.: How many adjustable numbers an AI model contains — the usual measure of its size. Modern language models have billions to trillions of them.
- tokentoken, n.: The small chunk of text an AI language model reads and writes — often a word, part of a word or a punctuation mark. AI usage, limits and prices are counted in tokens.
- data-constrained scalingdata-constrained scaling, n.: How to keep improving AI models when the supply of fresh, high-quality training text runs out — for example by reusing the same data several times, which works well at first and then stops helping.
Mentioned in
tokentoken, n.: The small chunk of text an AI language model reads and writes — often a word, part of a word or a punctuation mark. AI usage, limits and prices are counted in tokens., Chinchilla scaling lawsChinchilla scaling laws, n.: A 2022 DeepMind finding that, for a fixed amount of computing, language models had been made too big and trained on too little text; the best balance is roughly 20 word-pieces of text for every adjustable number inside the model., compute-optimal trainingcompute-optimal training, n.: Choosing a model's size and the amount of data it trains on so that a fixed amount of computing produces the best possible model — neither too big and undertrained nor too small and overfed., overtrainingovertraining, n.: Deliberately training a model on much more data than would be most efficient for its training budget, so that a smaller model reaches good quality and is cheaper to run afterwards., pretraining datapretraining data, n.: The enormous body of text, code and other material — mostly from the web — that a large AI model first learns from, before it is tuned to follow instructions or chat., data-constrained scalingdata-constrained scaling, n.: How to keep improving AI models when the supply of fresh, high-quality training text runs out — for example by reusing the same data several times, which works well at first and then stops helping.