Skip to content
Last Dictionary
Plane
Training & Post-trainingDeep LearningData & Datasets

annealing

nounAlso: anneal, annealing phase, annealing run, learning-rate annealing

In plain wordsThe last stretch of training, when the model's learning is slowed to a stop — often while it is fed a small helping of the best data — so it settles into its final, sharpest version.

  1. 1

    Pretrainingemerging

    The final phase of pretrainingpretraining, n.: The first and usually the most expensive stage of building a modern AI model, in which it learns general patterns from enormous amounts of raw text, images or code before being specialized for particular uses. in which the learning ratelearning rate, n.: How big a step a model takes each time it adjusts its settings during training. Too large and learning goes haywire; too small and it crawls. is decayed, often linearly to zero, while the data mixturedata mixture, n.: The recipe of different kinds of data a model is trained on — how much web text, code, books, math, science and other languages — which strongly shapes what the finished model is good at. shifts toward a smaller, higher-quality set — curated web text, mathematics, code, instruction-like and synthetic data — so that the model consolidates on it. Data seen while the learning rate falls has an outsized influence on the final model. Meta's Llama 3 report (2024) described annealing its largest model's learning rate to zero over the final 40 million tokens while upsampling very-high-quality sources, and judged small candidate datasets by annealing a half-trained 8-billion-parameter model on 40 billion tokens; AI2's OLMo 2 introduced a dedicated 'Dolmino' mix for the phase. It overlaps with, and is sometimes called, mid-trainingmid-training, n.: A middle stage in building a language model, between its broad initial training and its tuning into an assistant: it keeps learning the same way, but from a smaller, carefully chosen diet of math, code, reasoning or long documents..

    We annealed on 50 billion tokens of math, code and filtered web text.

  2. 2

    Learning-rate schedules

    Any gradual reduction of the learning rate during training, as in step decay, exponential decay or cosine annealingcosine annealing, n.: A popular plan for shrinking a model's step size during training: steps start large, shrink slowly, then faster, then level off near zero, following the shape of half a wave.: large steps early cover ground quickly, and small steps late allow fine convergence.

  3. 3

    Optimization and sampling

    Gradual lowering of a temperature parameter so that a search or a probability distribution moves from exploring to committing, as in simulated annealingsimulated annealing, n.: An optimization method inspired by slowly cooling metal: early on it sometimes accepts worse solutions, to escape dead ends, and becomes pickier as a 'temperature' setting gradually drops. or temperature annealing of a softmaxsoftmax, n.: A formula that turns a list of raw scores into probabilities — all positive and adding up to one — with the biggest share going to the highest score. or sampling distribution.

Origin

From metallurgy, where metal is heated and slowly cooled to remove defects. Scott Kirkpatrick, C. Daniel Gelatt and Mario Vecchi carried the word into optimization with simulated annealing (1983); 'learning-rate annealing' followed in neural-network training, and the data-shifting pretraining sense spread in 2024 through reports on MiniCPM, Llama 3 and OLMo 2.

Related

See also
  • mid-trainingmid-training, n.: A middle stage in building a language model, between its broad initial training and its tuning into an assistant: it keeps learning the same way, but from a smaller, carefully chosen diet of math, code, reasoning or long documents.
  • warmup-stable-decay schedulewarmup-stable-decay schedule, n.: A plan for how fast a model learns during training: speed up briefly, hold steady for most of the run, then slow down sharply at the end — which lets labs stop or extend training without starting over.
  • data mixturedata mixture, n.: The recipe of different kinds of data a model is trained on — how much web text, code, books, math, science and other languages — which strongly shapes what the finished model is good at.
  • learning rate schedulelearning rate schedule, n.: A plan for changing how big a model's learning steps are during training — usually starting cautiously, then taking big steps, then smaller and smaller ones as it settles in.
  • cosine annealingcosine annealing, n.: A popular plan for shrinking a model's step size during training: steps start large, shrink slowly, then faster, then level off near zero, following the shape of half a wave.
  • simulated annealingsimulated annealing, n.: An optimization method inspired by slowly cooling metal: early on it sometimes accepts worse solutions, to escape dead ends, and becomes pickier as a 'temperature' setting gradually drops.
Contrast with
  • simulated annealingsimulated annealing, n.: An optimization method inspired by slowly cooling metal: early on it sometimes accepts worse solutions, to escape dead ends, and becomes pickier as a 'temperature' setting gradually drops.

Sources

  1. Team OLMo, Walsh, P., Soldaini, L., Groeneveld, D. et al. (2024). 2 OLMo 2 Furious. arXiv:2501.00656. arxiv.org/abs/2501.00656v1
  2. Grattafiori, A. et al. (2024). The Llama 3 Herd of Models. arXiv:2407.21783.

Mentioned in

checkpointcheckpoint, n.: A saved snapshot of a model partway through or at the end of training, like a save point in a video game: training can resume from it after a crash, and finished snapshots are what people download and run., curriculum learningcurriculum learning, n.: Teaching a model the way schools teach people: starting with easy examples and gradually moving to harder ones, instead of presenting everything in random order., data mixturedata mixture, n.: The recipe of different kinds of data a model is trained on — how much web text, code, books, math, science and other languages — which strongly shapes what the finished model is good at., learning rate schedulelearning rate schedule, n.: A plan for changing how big a model's learning steps are during training — usually starting cautiously, then taking big steps, then smaller and smaller ones as it settles in., mid-trainingmid-training, n.: A middle stage in building a language model, between its broad initial training and its tuning into an assistant: it keeps learning the same way, but from a smaller, carefully chosen diet of math, code, reasoning or long documents., simulated annealingsimulated annealing, n.: An optimization method inspired by slowly cooling metal: early on it sometimes accepts worse solutions, to escape dead ends, and becomes pickier as a 'temperature' setting gradually drops., data-constrained scalingdata-constrained scaling, n.: How to keep improving AI models when the supply of fresh, high-quality training text runs out — for example by reusing the same data several times, which works well at first and then stops helping., warmup-stable-decay schedulewarmup-stable-decay schedule, n.: A plan for how fast a model learns during training: speed up briefly, hold steady for most of the run, then slow down sharply at the end — which lets labs stop or extend training without starting over.