Skip to content
Last Dictionary
Plane
Machine LearningMathematics & Statistics

ridge regression

nounAlso: Tikhonov regularization, L2-regularized regression, kernel ridge regression, ridge

In plain wordsLinear regression with a built-in brake: it fits the data while also keeping the weights small, which makes predictions more stable when there are many overlapping inputs.

  1. Linear regressionlinear regression, n.: Fitting a straight line (or flat surface) through data points to predict one number from others — say, a house's price from its floor area — choosing the line that keeps prediction errors smallest. estimated by minimizing the sum of squared residuals plus an L2 penalty λ‖β‖² on the coefficients, with the intercept normally left unpenalized. The solution exists in closed form even when XᵀX is singular — when features are collinear or outnumber observations — and shrinks every coefficient toward zero without setting any exactly to zero. Increasing λ raises bias and lowers variance; λ is chosen by cross-validationcross-validation, n.: A fair way to test a model when data are limited: split the data into several parts, train on all but one, test on the part left out, and rotate so every part takes a turn. or generalized cross-validation, with features standardized first because the penalty depends on their scale. Ridge is the MAP estimatemaximum a posteriori estimation, n.: Picking the single most believable setting of a model after combining the evidence with what was assumed beforehand. In practice it often amounts to ordinary training plus a penalty that keeps settings modest. under a zero-mean Gaussian priorprior, n.: What you believe, and how strongly, before looking at the evidence in front of you. In AI it also means built-in assumptions that steer what a system learns. on the coefficients and is a case of Tikhonov regularizationregularization, n.: Any technique that stops a model from memorizing its training examples too closely — often by penalizing complexity — so that it works better on new data.. Kernel ridge regression applies the kernel trickkernel trick, n.: A mathematical shortcut that lets simple straight-line methods find curved patterns, by computing similarities between examples as if the data had been mapped into a far richer space — without ever building that space. and, when λ equals the noise variance, gives the same predictive mean as a Gaussian processGaussian process, n.: A way of predicting an unknown curve or surface that gives not just a best guess at each point but an honest error bar, narrow near measured points and wide far from them. with the corresponding kernel.

    With 2,000 correlated gene-expression features and 300 patients, least squares failed but ridge regression gave stable coefficients.

β^ridge=arg⁡min⁡β∥y−Xβ∥22+λ∥β∥22=(X⊤X+λI)−1X⊤y\hat{\boldsymbol\beta}_{\text{ridge}} = \arg\min_{\boldsymbol\beta} \lVert \mathbf{y} - X\boldsymbol\beta \rVert_2^2 + \lambda \lVert \boldsymbol\beta \rVert_2^2 = (X^{\top}X + \lambda I)^{-1} X^{\top}\mathbf{y}

Origin

Arthur E. Hoerl applied his 'ridge analysis' of response surfaces to regression in 1962 and, with Robert W. Kennard, set out ridge regression in two 1970 Technometrics papers. 'Ridge' refers to the path traced by constrained optima in ridge analysis. The same mathematics had appeared in Andrey Tikhonov's work on ill-posed problems (from 1943) and David Phillips's 1962 paper.

Related

See also
  • regularizationregularization, n.: Any technique that stops a model from memorizing its training examples too closely — often by penalizing complexity — so that it works better on new data.
  • lassolasso, n.: A version of linear regression that keeps only the most useful inputs: it penalizes the total size of the weights so that many become exactly zero, leaving a simpler model.
  • elastic netelastic net, n.: A blend of two ways of keeping a model simple: it can drop useless inputs entirely while treating groups of similar inputs alike, instead of picking one of them at random.
  • linear regressionlinear regression, n.: Fitting a straight line (or flat surface) through data points to predict one number from others — say, a house's price from its floor area — choosing the line that keeps prediction errors smallest.
  • weight decayweight decay, n.: A training rule that shrinks every weight slightly at each step, like a small tax on large numbers. It keeps models simpler and usually helps them work better on new data.
  • kernel trickkernel trick, n.: A mathematical shortcut that lets simple straight-line methods find curved patterns, by computing similarities between examples as if the data had been mapped into a far richer space — without ever building that space.
  • Gaussian processGaussian process, n.: A way of predicting an unknown curve or surface that gives not just a best guess at each point but an honest error bar, narrow near measured points and wide far from them.
Contrast with
  • lassolasso, n.: A version of linear regression that keeps only the most useful inputs: it penalizes the total size of the weights so that many become exactly zero, leaving a simpler model.

Sources

  1. Hoerl, A. E. & Kennard, R. W. (1970). Ridge Regression: Biased Estimation for Nonorthogonal Problems. Technometrics, 12(1), 55–67. stat.cmu.edu/technometrics/70-79/VOL-12-01/v1201055.pdf

Mentioned in

Gaussian processGaussian process, n.: A way of predicting an unknown curve or surface that gives not just a best guess at each point but an honest error bar, narrow near measured points and wide far from them., kernel trickkernel trick, n.: A mathematical shortcut that lets simple straight-line methods find curved patterns, by computing similarities between examples as if the data had been mapped into a far richer space — without ever building that space., lassolasso, n.: A version of linear regression that keeps only the most useful inputs: it penalizes the total size of the weights so that many become exactly zero, leaving a simpler model., linear regressionlinear regression, n.: Fitting a straight line (or flat surface) through data points to predict one number from others — say, a house's price from its floor area — choosing the line that keeps prediction errors smallest., normnorm, n.: A way of measuring the size, or length, of a list of numbers — for example the straight-line distance from the origin to the point the list describes., ordinary least squaresordinary least squares, n.: The standard way to fit a line through data: choose the line that makes the total of the squared vertical distances from the points to the line as small as possible., regularizationregularization, n.: Any technique that stops a model from memorizing its training examples too closely — often by penalizing complexity — so that it works better on new data., weight decayweight decay, n.: A training rule that shrinks every weight slightly at each step, like a small tax on large numbers. It keeps models simpler and usually helps them work better on new data., elastic netelastic net, n.: A blend of two ways of keeping a model simple: it can drop useless inputs entirely while treating groups of similar inputs alike, instead of picking one of them at random., maximum a posteriori estimationmaximum a posteriori estimation, n.: Picking the single most believable setting of a model after combining the evidence with what was assumed beforehand. In practice it often amounts to ordinary training plus a penalty that keeps settings modest.