Skip to content
Last Dictionary
Plane
Machine LearningDeep LearningMathematics & Statistics

regularization

nounAlso: regularisation, regularizer, regularize, penalty term, L1 regularization, L2 regularization, shrinkage

In plain wordsAny technique that stops a model from memorizing its training examples too closely — often by penalizing complexity — so that it works better on new data.

  1. 1

    Machine learning

    A constraint, penalty or training practice that discourages a model from fitting the quirks of its particular training sample, so that it does better on data it has not seen (generalizationgeneralization, n.: How well a model copes with new examples it has never seen, rather than just memorizing its training data, like a student who understands the method, not just the answers.), usually by limiting or penalizing model complexity. In the classic form a penalty is added to the training objective: an L2 penalty on the squared size of the weights, as in ridge regressionridge regression, n.: Linear regression with a built-in brake: it fits the data while also keeping the weights small, which makes predictions more stable when there are many overlapping inputs. and weight decayweight decay, n.: A training rule that shrinks every weight slightly at each step, like a small tax on large numbers. It keeps models simpler and usually helps them work better on new data., shrinks them smoothly, while an L1 penalty, as in the lassolasso, n.: A version of linear regression that keeps only the most useful inputs: it penalizes the total size of the weights so that many become exactly zero, leaving a simpler model., drives many to exactly zero and yields sparse models. The strength λ is a hyperparameterhyperparameter, n.: A setting chosen by people before training, such as how fast a model learns or how big it is, as opposed to the internal values the model learns by itself from data. tuned by cross-validationcross-validation, n.: A fair way to test a model when data are limited: split the data into several parts, train on all but one, test on the part left out, and rotate so every part takes a turn.. In Bayesian terms such penalties correspond to priorsprior, n.: What you believe, and how strongly, before looking at the evidence in front of you. In AI it also means built-in assumptions that steer what a system learns. on the weights, and the penalized optimum is the MAP estimatemaximum a posteriori estimation, n.: Picking the single most believable setting of a model after combining the evidence with what was assumed beforehand. In practice it often amounts to ordinary training plus a penalty that keeps settings modest.. In deep learning the word also covers dropoutdropout, n.: A training trick that randomly switches off a fraction of a network's neurons at each step, so the network cannot lean too heavily on any one of them and learns sturdier patterns., early stoppingearly stopping, n.: Stopping training at the point where the model does best on practice data it has not learned from, before it starts memorizing quirks of its training examples., data augmentationdata augmentation, n.: Creating extra training examples by changing existing ones in ways that keep their meaning — a flipped, cropped or recolored photo of a cat still shows a cat — so the model learns more robustly., label smoothinglabel smoothing, n.: A training trick that tells a model the correct answer is very likely rather than absolutely certain, which stops it from becoming overconfident and often helps it generalize. and the implicit regularizationimplicit regularization, n.: The hidden preferences of a training method itself: even with no explicit rule against complexity, the way a model is trained tends to steer it toward simpler solutions. of the optimizer. All of them accept a little extra bias for a larger cut in variance (see bias–variance tradeoffbias–variance tradeoff, n.: The balancing act between models too simple to capture a pattern and models so flexible that they also learn the noise. The best predictions usually come from somewhere in between.).

    Validation loss stopped rising once we doubled the L2 regularization and added dropout.

  2. 2

    Mathematics

    A method for making an ill-posed problem — one whose solution is not unique or changes wildly with small changes in the data, as in many inverse problems such as image deblurring — well-posed by adding a constraint or penalty, as in Tikhonov regularization. Ridge regressionridge regression, n.: Linear regression with a built-in brake: it fits the data while also keeping the weights small, which makes predictions more stable when there are many overlapping inputs. is this idea applied to least squares.

w^=arg⁡min⁡w  1n∑i=1nL(yi,fw(xi))+λ Ω(w)\hat{\mathbf{w}} = \arg\min_{\mathbf{w}} \; \frac{1}{n}\sum_{i=1}^{n} L\big(y_i, f_{\mathbf{w}}(\mathbf{x}_i)\big) + \lambda\, \Omega(\mathbf{w})

Origin

Andrey Tikhonov studied the stabilization of ill-posed inverse problems from 1943 and named a 'regularization method' in 1963 papers. In statistics, Arthur Hoerl and Robert Kennard's ridge regression (1970) and Robert Tibshirani's lasso (1996) made penalized estimation standard; neural-network researchers took up the idea, notably as weight decay, from the late 1980s.

Usage note

Usage varies: some reserve the word for explicit penalty terms, while others, following Goodfellow, Bengio and Courville's 'Deep Learning' (2016), apply it to any change aimed at test rather than training error. The intercept or bias term is usually left unpenalized.

Related

See also
  • overfittingoverfitting, n.: When a model learns its training examples too well — quirks and noise included — so it does great on data it has seen but poorly on anything new, like a student who memorizes answers without understanding.
  • ridge regressionridge regression, n.: Linear regression with a built-in brake: it fits the data while also keeping the weights small, which makes predictions more stable when there are many overlapping inputs.
  • lassolasso, n.: A version of linear regression that keeps only the most useful inputs: it penalizes the total size of the weights so that many become exactly zero, leaving a simpler model.
  • elastic netelastic net, n.: A blend of two ways of keeping a model simple: it can drop useless inputs entirely while treating groups of similar inputs alike, instead of picking one of them at random.
  • weight decayweight decay, n.: A training rule that shrinks every weight slightly at each step, like a small tax on large numbers. It keeps models simpler and usually helps them work better on new data.
  • dropoutdropout, n.: A training trick that randomly switches off a fraction of a network's neurons at each step, so the network cannot lean too heavily on any one of them and learns sturdier patterns.
  • early stoppingearly stopping, n.: Stopping training at the point where the model does best on practice data it has not learned from, before it starts memorizing quirks of its training examples.
  • bias–variance tradeoffbias–variance tradeoff, n.: The balancing act between models too simple to capture a pattern and models so flexible that they also learn the noise. The best predictions usually come from somewhere in between.

Sources

  1. Tikhonov, A. N. (1963). Solution of Incorrectly Formulated Problems and the Regularization Method. Soviet Mathematics Doklady, 4, 1035–1038.
  2. Hoerl, A. E. & Kennard, R. W. (1970). Ridge Regression: Biased Estimation for Nonorthogonal Problems. Technometrics, 12(1), 55–67. stat.cmu.edu/technometrics/70-79/VOL-12-01/v1201055.pdf

Mentioned in

loss functionloss function, n.: A score that measures how wrong a model's predictions are. Training means adjusting the model, step by step, to push this number down., overfittingoverfitting, n.: When a model learns its training examples too well — quirks and noise included — so it does great on data it has seen but poorly on anything new, like a student who memorizes answers without understanding., AdamWAdamW, prop. n.: A corrected version of the popular Adam training method that applies its steady pull toward smaller weights separately from the main update. It is the standard choice for training large language models., bias–variance tradeoffbias–variance tradeoff, n.: The balancing act between models too simple to capture a pattern and models so flexible that they also learn the noise. The best predictions usually come from somewhere in between., curse of dimensionalitycurse of dimensionality, n.: The problems that pile up when each item is described by very many measurements: even huge datasets end up thinly scattered across the possibilities, and 'near' and 'far' stop meaning much., data augmentationdata augmentation, n.: Creating extra training examples by changing existing ones in ways that keep their meaning — a flipped, cropped or recolored photo of a cat still shows a cat — so the model learns more robustly., dropoutdropout, n.: A training trick that randomly switches off a fraction of a network's neurons at each step, so the network cannot lean too heavily on any one of them and learns sturdier patterns., early stoppingearly stopping, n.: Stopping training at the point where the model does best on practice data it has not learned from, before it starts memorizing quirks of its training examples., empirical risk minimizationempirical risk minimization, n.: The basic recipe of most machine learning: from the models on offer, pick the one with the smallest errors on the training examples, trusting that score as a guide to how it will do on new data., feature scalingfeature scaling, n.: Putting input measurements on comparable scales, so that income in dollars and age in years, say, both span similar ranges and no feature dominates just because of its units., generalized additive modelgeneralized additive model, n.: A prediction model that adds up separate, possibly curved effects of each input, so each input's influence can be drawn as its own graph and read off directly., generalized linear modelgeneralized linear model, n.: A family of statistical models that extends straight-line regression to outcomes such as yes/no answers, counts or waiting times, by linking the average outcome to a weighted sum of inputs through a suitable curve., hyperparameterhyperparameter, n.: A setting chosen by people before training, such as how fast a model learns or how big it is, as opposed to the internal values the model learns by itself from data., inductive biasinductive bias, n.: The assumptions built into a learning method that decide how it fills the gaps between the examples it was shown, such as that similar inputs get similar answers. Every learner needs some to cope with new cases., lassolasso, n.: A version of linear regression that keeps only the most useful inputs: it penalizes the total size of the weights so that many become exactly zero, leaving a simpler model., learning curvelearning curve, n.: A graph showing how a model's performance improves as it gets more training — more examples or more time. Its shape tells you whether more data or a bigger model would help., logistic regressionlogistic regression, n.: A simple, widely used method for yes-or-no predictions that outputs a probability — say, a 70% chance a customer cancels — by passing a weighted sum of inputs through an S-shaped curve., matrix factorizationmatrix factorization, n.: Breaking a big table of numbers, such as which users liked which films, into two smaller tables of hidden traits, so missing entries can be filled in by multiplying them back together., model capacitymodel capacity, n.: How much a model is able to learn: the variety and complexity of patterns it can represent or memorize. Too little and it misses real patterns; very large models can even memorize pure noise., normnorm, n.: A way of measuring the size, or length, of a list of numbers — for example the straight-line distance from the origin to the point the list describes., Occam's razorOccam's razor, n.: A rule of thumb for choosing between explanations: if two fit the evidence equally well, bet on the one that assumes less. In AI it motivates preferring simpler models, which are less likely to fit noise., polynomial regressionpolynomial regression, n.: Fitting a curve instead of a straight line by also using the square, cube and higher powers of an input. Higher powers let the curve bend more but easily start chasing noise., priorprior, n.: What you believe, and how strongly, before looking at the evidence in front of you. In AI it also means built-in assumptions that steer what a system learns., ridge regressionridge regression, n.: Linear regression with a built-in brake: it fits the data while also keeping the weights small, which makes predictions more stable when there are many overlapping inputs.