Skip to content
Last Dictionary
Plane
Machine LearningMathematics & Statistics

elastic net

nounAlso: elastic-net regularization, elastic net regression

In plain wordsA blend of two ways of keeping a model simple: it can drop useless inputs entirely while treating groups of similar inputs alike, instead of picking one of them at random.

  1. A penalized regression method whose penalty is a weighted mix of the lassolasso, n.: A version of linear regression that keeps only the most useful inputs: it penalizes the total size of the weights so that many become exactly zero, leaving a simpler model.'s sum of absolute coefficients (L1) and ridge regressionridge regression, n.: Linear regression with a built-in brake: it fits the data while also keeping the weights small, which makes predictions more stable when there are many overlapping inputs.'s sum of squared coefficients (L2), the balance being set by a mixing weight (see regularizationregularization, n.: Any technique that stops a model from memorizing its training examples too closely — often by penalizing complexity — so that it works better on new data.); in the common glmnet parametrization a mixing weight α = 1 gives the lasso and α = 0 gives ridge. The L1 part produces sparse solutions; the L2 part makes the objective strictly convex, stabilizes the solution path and produces a grouping effect, so strongly correlated features tend to enter or leave the model together with similar coefficients. Unlike the lasso, it can select more features than there are observations. The overall strength λ and the mix α are tuned by cross-validationcross-validation, n.: A fair way to test a model when data are limited: split the data into several parts, train on all but one, test on the part left out, and rotate so every part takes a turn.. It is widely used in high-dimensional problems such as genomics, and the same penalty is applied to logistic regressionlogistic regression, n.: A simple, widely used method for yes-or-no predictions that outputs a probability — say, a 70% chance a customer cancels — by passing a weighted sum of inputs through an S-shaped curve. and other generalized linear modelsgeneralized linear model, n.: A family of statistical models that extends straight-line regression to outcomes such as yes/no answers, counts or waiting times, by linking the average outcome to a weighted sum of inputs through a suitable curve..

    Because the genetic markers come in correlated blocks, we used an elastic net with alpha 0.5 instead of the plain lasso.

β^=arg⁡min⁡β12n∥y−Xβ∥22+λ(α∥β∥1+1−α2∥β∥22)\hat{\boldsymbol\beta} = \arg\min_{\boldsymbol\beta} \frac{1}{2n}\lVert \mathbf{y} - X\boldsymbol\beta \rVert_2^2 + \lambda\Big(\alpha \lVert \boldsymbol\beta \rVert_1 + \frac{1-\alpha}{2}\lVert \boldsymbol\beta \rVert_2^2\Big)

Origin

Introduced by Hui Zou and Trevor Hastie in 'Regularization and Variable Selection via the Elastic Net' (Journal of the Royal Statistical Society, Series B, 2005), who compared it to a stretchable fishing net that keeps all the big fish. The coordinate-descent package glmnet by Jerome Friedman, Trevor Hastie and Robert Tibshirani, described in a 2010 paper, made it a default tool.

Related

See also
  • lassolasso, n.: A version of linear regression that keeps only the most useful inputs: it penalizes the total size of the weights so that many become exactly zero, leaving a simpler model.
  • ridge regressionridge regression, n.: Linear regression with a built-in brake: it fits the data while also keeping the weights small, which makes predictions more stable when there are many overlapping inputs.
  • regularizationregularization, n.: Any technique that stops a model from memorizing its training examples too closely — often by penalizing complexity — so that it works better on new data.
  • feature selectionfeature selection, n.: Choosing which of the available input measurements a model should use and dropping the rest, so the model becomes simpler, faster and often more accurate.
  • generalized linear modelgeneralized linear model, n.: A family of statistical models that extends straight-line regression to outcomes such as yes/no answers, counts or waiting times, by linking the average outcome to a weighted sum of inputs through a suitable curve.
  • cross-validationcross-validation, n.: A fair way to test a model when data are limited: split the data into several parts, train on all but one, test on the part left out, and rotate so every part takes a turn.

Sources

  1. Zou, H. & Hastie, T. (2005). Regularization and Variable Selection via the Elastic Net. Journal of the Royal Statistical Society, Series B, 67(2), 301–320. ideas.repec.org/a/bla/jorssb/v67y2005i2p301-320.html

Mentioned in

lassolasso, n.: A version of linear regression that keeps only the most useful inputs: it penalizes the total size of the weights so that many become exactly zero, leaving a simpler model., linear regressionlinear regression, n.: Fitting a straight line (or flat surface) through data points to predict one number from others — say, a house's price from its floor area — choosing the line that keeps prediction errors smallest., regularizationregularization, n.: Any technique that stops a model from memorizing its training examples too closely — often by penalizing complexity — so that it works better on new data., ridge regressionridge regression, n.: Linear regression with a built-in brake: it fits the data while also keeping the weights small, which makes predictions more stable when there are many overlapping inputs., Jerome FriedmanJerome Friedman, prop. n.: An American statistician who invented gradient boosting, the method behind XGBoost and other leading tools for spreadsheet-style data, and co-wrote a standard machine-learning textbook.