Skip to content
Last Dictionary
Plane
Training & Post-trainingLanguage Models & TransformersHistory & Landmarks

Chinchilla scaling laws

nounAlso: Chinchilla scaling, Chinchilla law, Hoffmann scaling laws, Chinchilla

In plain wordsA 2022 DeepMind finding that, for a fixed amount of computing, language models had been made too big and trained on too little text; the best balance is roughly 20 word-pieces of text for every adjustable number inside the model.

  1. The compute-optimal scaling relations of Hoffmann et al. (2022), which found that the loss-minimizing parameter count N and token count D for a compute budgetcompute budget, n.: The total amount of computing a team can spend on training (or running) a model, set in advance like a money budget and used to decide how big the model should be and how much data it sees. C grow in equal proportion: N_opt ∝ Cᵃ and D_opt ∝ Cᵇ with a ≈ b ≈ 0.5. The evidence came from more than 400 language models, the smallest with 70 million parameters and the largest with over 16 billion, trained on between 5 and 500 billion tokens, and three estimation methods broadly agreed: varying the token count at fixed model sizes, IsoFLOP profilesIsoFLOP analysis, n.: A way of finding the best model size for a given amount of computing: train several models of different sizes at exactly the same computing cost and see which one ends up best., and fitting the parametric loss below. The practical upshot, about 20 tokens per parametertokens per parameter, n.: How many pieces of text a language model reads during training for each adjustable number inside it — a quick way to tell whether a model was trained on a little or a lot of data for its size. for dense transformers, contradicted the earlier OpenAI recommendation (Kaplan et al., 2020) to grow models much faster than data (N ∝ C^0.73) and implied that GPT-3GPT-3, prop. n.: A huge 2020 OpenAI language model that could perform new tasks simply from instructions or a few examples written into its prompt, without retraining. It made prompting a new way of programming., Gopher and Megatron-Turing NLG were undertrained. The 70-billion-parameter Chinchilla, trained on 1.4 trillion tokens with the compute of the 280-billion-parameter Gopher, outperformed it. A 2024 replication by Epoch AIEpoch AI, prop. n.: A research nonprofit that gathers data on how fast AI is advancing — the computing power, data and money behind it — and publishes forecasts and difficult tests for AI. researchers found the published parametric fit inconsistent with the other two methods; refitted, it supports the 20:1 ratio.

    By Chinchilla scaling laws, a 10²⁴ FLOP budget calls for about 90 billion parameters and 1.8 trillion tokens.

L(N,D)=E+ANα+BDβ,E≈1.69,  α≈0.34,  β≈0.28L(N, D) = E + \frac{A}{N^{\alpha}} + \frac{B}{D^{\beta}}, \qquad E \approx 1.69,\; \alpha \approx 0.34,\; \beta \approx 0.28

Origin

Introduced by Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch and colleagues at DeepMind in 'Training Compute-Optimal Large Language Models' (arXiv, March 2022; at NeurIPS 2022 as 'An Empirical Analysis of Compute-Optimal Large Language Model Training'), named after the model built to test them. Tamay Besiroglu and colleagues published the replication in April 2024.

Usage note

'Chinchilla-optimal' means optimal for training compute only; most models released since 2023 are deliberately trained on far more tokens (overtrainingovertraining, n.: Deliberately training a model on much more data than would be most efficient for its training budget, so that a smaller model reaches good quality and is cheaper to run afterwards.) because smaller models are cheaper to serve. The 20:1 ratio reflects Chinchilla's data, tokenizer and architecture, not a universal constant.

Related

See also
  • scaling lawsscaling laws, n.: Measured patterns showing that an AI model's error shrinks smoothly and predictably as the model is made bigger and given more data and computing power, so labs can forecast a giant model's performance from small, cheap experiments.
  • compute-optimal trainingcompute-optimal training, n.: Choosing a model's size and the amount of data it trains on so that a fixed amount of computing produces the best possible model — neither too big and undertrained nor too small and overfed.
  • tokens per parametertokens per parameter, n.: How many pieces of text a language model reads during training for each adjustable number inside it — a quick way to tell whether a model was trained on a little or a lot of data for its size.
  • IsoFLOP analysisIsoFLOP analysis, n.: A way of finding the best model size for a given amount of computing: train several models of different sizes at exactly the same computing cost and see which one ends up best.
  • overtrainingovertraining, n.: Deliberately training a model on much more data than would be most efficient for its training budget, so that a smaller model reaches good quality and is cheaper to run afterwards.
  • irreducible lossirreducible loss, n.: The part of a model's error that no amount of extra size, data or training can remove, because the thing being predicted is itself partly random — like the unpredictability of the next word in an ordinary sentence.
Contrast with
  • overtrainingovertraining, n.: Deliberately training a model on much more data than would be most efficient for its training budget, so that a smaller model reaches good quality and is cheaper to run afterwards.

Sources

  1. Hoffmann, J., Borgeaud, S., Mensch, A. et al. (2022). Training Compute-Optimal Large Language Models. arXiv:2203.15556. arxiv.org/abs/2203.15556v1
  2. Besiroglu, T., Erdil, E., Barnett, M. & You, J. (2024). Chinchilla Scaling: A replication attempt. arXiv:2404.10102. arxiv.org/abs/2404.10102

Mentioned in

scaling lawsscaling laws, n.: Measured patterns showing that an AI model's error shrinks smoothly and predictably as the model is made bigger and given more data and computing power, so labs can forecast a giant model's performance from small, cheap experiments., compute budgetcompute budget, n.: The total amount of computing a team can spend on training (or running) a model, set in advance like a money budget and used to decide how big the model should be and how much data it sees., compute-optimal trainingcompute-optimal training, n.: Choosing a model's size and the amount of data it trains on so that a fixed amount of computing produces the best possible model — neither too big and undertrained nor too small and overfed., cosine annealingcosine annealing, n.: A popular plan for shrinking a model's step size during training: steps start large, shrink slowly, then faster, then level off near zero, following the shape of half a wave., data walldata wall, n.: The worry that AI companies will run out of new human-written text to train ever-larger models on, because the useful public internet has already been largely used up., DeepMindDeepMind, prop. n.: A London artificial intelligence company, bought by Google in 2014, famous for programs that mastered the game of Go and predicted the shapes of proteins. In 2023 it merged into Google DeepMind., Jared KaplanJared Kaplan, prop. n.: An American physicist who showed that language models improve predictably as they grow and train on more data, and who co-founded the AI company Anthropic., overtrainingovertraining, n.: Deliberately training a model on much more data than would be most efficient for its training budget, so that a smaller model reaches good quality and is cheaper to run afterwards., parameter countparameter count, n.: How many adjustable numbers an AI model contains — the usual measure of its size. Modern language models have billions to trillions of them., power lawpower law, n.: A rule linking two quantities in which scaling one up by a given factor always changes the other by the same multiple, wherever you start. Plotted on logarithmic axes, it is a straight line., tokens per parametertokens per parameter, n.: How many pieces of text a language model reads during training for each adjustable number inside it — a quick way to tell whether a model was trained on a little or a lot of data for its size., data-constrained scalingdata-constrained scaling, n.: How to keep improving AI models when the supply of fresh, high-quality training text runs out — for example by reusing the same data several times, which works well at first and then stops helping., irreducible lossirreducible loss, n.: The part of a model's error that no amount of extra size, data or training can remove, because the thing being predicted is itself partly random — like the unpredictability of the next word in an ordinary sentence., IsoFLOP analysisIsoFLOP analysis, n.: A way of finding the best model size for a given amount of computing: train several models of different sizes at exactly the same computing cost and see which one ends up best., PaLMPaLM, prop. n.: A very large Google language model from 2022 that showed sudden jumps in reasoning, and even explained jokes, as it grew. Its successor powered Google's Bard chatbot before the Gemini models replaced it.