overtraining
In plain wordsDeliberately training a model on much more data than would be most efficient for its training budget, so that a smaller model reaches good quality and is cheaper to run afterwards.
- 1
Scaling
Training a model on more tokens than is compute-optimalcompute-optimal training, n.: Choosing a model's size and the amount of data it trains on so that a fixed amount of computing produces the best possible model — neither too big and undertrained nor too small and overfed. for its size, accepting a higher training cost for a given loss in exchange for a smaller model that is cheaper and faster at inference. Because loss keeps falling with more tokens, an overtrained small model can match a compute-optimal larger one, and over a model's deployed life serving costs usually dominate, so total cost falls. Overtraining is described by a multiplier over the Chinchilla token count — Samir Yitzhak Gadre and colleagues (2024) studied runs 32× beyond it — or by tokens per parametertokens per parameter, n.: How many pieces of text a language model reads during training for each adjustable number inside it — a quick way to tell whether a model was trained on a little or a lot of data for its size.. Nikhil Sardana and colleagues (2023) extended the Chinchilla scaling lawsChinchilla scaling laws, n.: A 2022 DeepMind finding that, for a fixed amount of computing, language models had been made too big and trained on too little text; the best balance is roughly 20 word-pieces of text for every adjustable number inside the model. to include expected inference demand and found quality still improving at 10,000 tokens per parameter. Since LLaMA (2023), most open-weight models have been trained this way.
We overtrained the 3B model on 12 trillion tokens because it will serve billions of requests.
- 2
Older usagehistorical
Training a neural network for so long on a fixed training set that it fits noise and generalizes worse; a synonym for overfittingoverfitting, n.: When a model learns its training examples too well — quirks and noise included — so it does great on data it has seen but poorly on anything new, like a student who memorizes answers without understanding. common in neural-network literature of the late 1980s and 1990s, and the motivation for early stoppingearly stopping, n.: Stopping training at the point where the model does best on practice data it has not learned from, before it starts memorizing quirks of its training examples..
Origin
The early-stopping sense is from neural-network work of the late 1980s and 1990s. The scaling sense followed the Chinchilla paper (2022): Meta's LLaMA (February 2023) trained 7- and 13-billion-parameter models on a trillion tokens, and 'over-training' was analyzed by Nikhil Sardana and colleagues (December 2023) and Samir Yitzhak Gadre and colleagues (March 2024).
Usage note
The two senses point in opposite directions: in the modern sense extra training on fresh data improves the model; in the older sense extra passes over the same data harm it. Distinct from overfittingoverfitting, n.: When a model learns its training examples too well — quirks and noise included — so it does great on data it has seen but poorly on anything new, like a student who memorizes answers without understanding., which concerns generalization rather than how a budget is allocated.
Related
- compute-optimal trainingcompute-optimal training, n.: Choosing a model's size and the amount of data it trains on so that a fixed amount of computing produces the best possible model — neither too big and undertrained nor too small and overfed.
- tokens per parametertokens per parameter, n.: How many pieces of text a language model reads during training for each adjustable number inside it — a quick way to tell whether a model was trained on a little or a lot of data for its size.
- Chinchilla scaling lawsChinchilla scaling laws, n.: A 2022 DeepMind finding that, for a fixed amount of computing, language models had been made too big and trained on too little text; the best balance is roughly 20 word-pieces of text for every adjustable number inside the model.
- inference costinference cost, n.: What it costs to use an AI model after it has been built: the computing power, electricity and hardware needed every time it answers, usually billed per chunk of text.
- small language modelsmall language model, n.: A language model built to be compact enough to run cheaply and quickly, even on a laptop or phone, trading some knowledge and skill for speed, privacy and lower cost.
- data-constrained scalingdata-constrained scaling, n.: How to keep improving AI models when the supply of fresh, high-quality training text runs out — for example by reusing the same data several times, which works well at first and then stops helping.
- overfittingoverfitting, n.: When a model learns its training examples too well — quirks and noise included — so it does great on data it has seen but poorly on anything new, like a student who memorizes answers without understanding.
- compute-optimal trainingcompute-optimal training, n.: Choosing a model's size and the amount of data it trains on so that a fixed amount of computing produces the best possible model — neither too big and undertrained nor too small and overfed.
Sources
- Sardana, N., Portes, J., Doubov, S. & Frankle, J. (2023). Beyond Chinchilla-Optimal: Accounting for Inference in Language Model Scaling Laws. arXiv:2401.00448. arxiv.org/abs/2401.00448
- Gadre, S. Y. et al. (2024). Language models scale reliably with over-training and on downstream tasks. arXiv:2403.08540. arxiv.org/abs/2403.08540
Mentioned in
Chinchilla scaling lawsChinchilla scaling laws, n.: A 2022 DeepMind finding that, for a fixed amount of computing, language models had been made too big and trained on too little text; the best balance is roughly 20 word-pieces of text for every adjustable number inside the model., compute budgetcompute budget, n.: The total amount of computing a team can spend on training (or running) a model, set in advance like a money budget and used to decide how big the model should be and how much data it sees., compute-optimal trainingcompute-optimal training, n.: Choosing a model's size and the amount of data it trains on so that a fixed amount of computing produces the best possible model — neither too big and undertrained nor too small and overfed., convergenceconvergence, n.: The point at which training settles down and further steps no longer meaningfully improve the model — and, in theory, a guarantee that a method will reach a solution, and how fast., data walldata wall, n.: The worry that AI companies will run out of new human-written text to train ever-larger models on, because the useful public internet has already been largely used up., small language modelsmall language model, n.: A language model built to be compact enough to run cheaply and quickly, even on a laptop or phone, trading some knowledge and skill for speed, privacy and lower cost., tokens per parametertokens per parameter, n.: How many pieces of text a language model reads during training for each adjustable number inside it — a quick way to tell whether a model was trained on a little or a lot of data for its size.