weight decay
In plain wordsA training rule that shrinks every weight slightly at each step, like a small tax on large numbers. It keeps models simpler and usually helps them work better on new data.
A regularizationregularization, n.: Any technique that stops a model from memorizing its training examples too closely — often by penalizing complexity — so that it works better on new data. technique that multiplies each weight by a factor slightly below one at every update (equivalently, subtracts a small multiple λ of the weight), pulling parameters toward zero unless the lossloss function, n.: A score that measures how wrong a model's predictions are. Training means adjusting the model, step by step, to push this number down. gradient holds them away. For plain stochastic gradient descentstochastic gradient descent, n.: A faster version of gradient descent that estimates which way is downhill from a small random sample of the training data at each step, instead of checking the whole dataset. it is equivalent to adding an L2 penalty (λ/2)‖θ‖² to the loss, as in ridge regressionridge regression, n.: Linear regression with a built-in brake: it fits the data while also keeping the weights small, which makes predictions more stable when there are many overlapping inputs.; for adaptive optimizers it is not, which is why AdamWAdamW, prop. n.: A corrected version of the popular Adam training method that applies its steady pull toward smaller weights separately from the main update. It is the standard choice for training large language models. applies it in decoupled form. In networks with normalization layersbatch normalization, n.: A layer that rescales the numbers flowing through a network so that, across each batch of training examples, they keep a steady average and spread. It makes deep networks train faster and more reliably. many weights are scale-invariant, so weight decay acts less as a capacity limit than as a control on the effective learning rate, holding weight norms in equilibrium. It shapes training dynamics in other ways too: it is a key ingredient of grokkinggrokking, n.: When a model that has merely memorized its training examples suddenly, long afterward, figures out the underlying rule and starts getting new examples right. and affects stability and plasticity in long runs. Typical values range from about 10⁻⁴ (SGD on vision models) to 0.1 (AdamW on language models), and biases, normalization gains and often embeddings are exempted.
Raising weight decay from 0.01 to 0.1 closed most of the gap between training and validation loss.
Origin
Penalizing large weights to improve generalization appears in the late-1980s connectionist literature (for example Stephen Hanson and Lorien Pratt, 1988), and Anders Krogh and John Hertz analyzed it in 'A Simple Weight Decay Can Improve Generalization' (NeurIPS 1991). The underlying L2 penalty goes back to Andrey Tikhonov's regularization of ill-posed problems (1963) and to ridge regression (Arthur Hoerl and Robert Kennard, 1970).
Usage note
Often conflated with L2 regularization; the two coincide only for plain SGD. Not to be confused with learning-rate decay, which shrinks step sizes rather than weights.
Related
- AdamWAdamW, prop. n.: A corrected version of the popular Adam training method that applies its steady pull toward smaller weights separately from the main update. It is the standard choice for training large language models.
- regularizationregularization, n.: Any technique that stops a model from memorizing its training examples too closely — often by penalizing complexity — so that it works better on new data.
- ridge regressionridge regression, n.: Linear regression with a built-in brake: it fits the data while also keeping the weights small, which makes predictions more stable when there are many overlapping inputs.
- overfittingoverfitting, n.: When a model learns its training examples too well — quirks and noise included — so it does great on data it has seen but poorly on anything new, like a student who memorizes answers without understanding.
- grokkinggrokking, n.: When a model that has merely memorized its training examples suddenly, long afterward, figures out the underlying rule and starts getting new examples right.
- implicit regularizationimplicit regularization, n.: The hidden preferences of a training method itself: even with no explicit rule against complexity, the way a model is trained tends to steer it toward simpler solutions.
- dropoutdropout, n.: A training trick that randomly switches off a fraction of a network's neurons at each step, so the network cannot lean too heavily on any one of them and learns sturdier patterns.
- learning rate schedulelearning rate schedule, n.: A plan for changing how big a model's learning steps are during training — usually starting cautiously, then taking big steps, then smaller and smaller ones as it settles in.
Sources
- Krogh, A. & Hertz, J. A. (1992). A Simple Weight Decay Can Improve Generalization. Advances in Neural Information Processing Systems 4.
- Loshchilov, I. & Hutter, F. (2019). Decoupled Weight Decay Regularization. ICLR 2019. arxiv.org/abs/1711.05101
Mentioned in
AdamWAdamW, prop. n.: A corrected version of the popular Adam training method that applies its steady pull toward smaller weights separately from the main update. It is the standard choice for training large language models., biasbias, n.: A word with several AI meanings: an adjustable number added inside a model; a systematic error in estimates; or unfair treatment of some groups of people by an AI system., dropoutdropout, n.: A training trick that randomly switches off a fraction of a network's neurons at each step, so the network cannot lean too heavily on any one of them and learns sturdier patterns., MuonMuon, prop. n.: A newer training method for neural networks that tidies each update so it pushes evenly in all directions within a weight table. Several groups report that it trains language models with less computing than the standard method., normnorm, n.: A way of measuring the size, or length, of a list of numbers — for example the straight-line distance from the origin to the point the list describes., priorprior, n.: What you believe, and how strongly, before looking at the evidence in front of you. In AI it also means built-in assumptions that steer what a system learns., regularizationregularization, n.: Any technique that stops a model from memorizing its training examples too closely — often by penalizing complexity — so that it works better on new data., ridge regressionridge regression, n.: Linear regression with a built-in brake: it fits the data while also keeping the weights small, which makes predictions more stable when there are many overlapping inputs., weightsweights, n.: The learned numbers inside a neural network that set how strongly each part influences the next. The finished set of weights is, in effect, the trained model itself., grokkinggrokking, n.: When a model that has merely memorized its training examples suddenly, long afterward, figures out the underlying rule and starts getting new examples right., LionLion, prop. n.: A training method discovered by a computer search over possible algorithms. It moves every setting by the same small amount at each step, choosing only the direction, and needs less memory than the popular Adam method., maximum a posteriori estimationmaximum a posteriori estimation, n.: Picking the single most believable setting of a model after combining the evidence with what was assumed beforehand. In practice it often amounts to ordinary training plus a penalty that keeps settings modest.