Skip to content
Last Dictionary
Plane
Machine LearningMathematics & Statistics

lasso

nounAlso: least absolute shrinkage and selection operator, L1-penalized regression, lasso regression, basis pursuit denoising

In plain wordsA version of linear regression that keeps only the most useful inputs: it penalizes the total size of the weights so that many become exactly zero, leaving a simpler model.

  1. Linear regressionlinear regression, n.: Fitting a straight line (or flat surface) through data points to predict one number from others — say, a house's price from its floor area — choosing the line that keeps prediction errors smallest., or another model, estimated with an L1 penalty λΣ|βⱼ| on the coefficients — equivalently, by minimizing squared error subject to a bound on the sum of their absolute values. Because the L1 constraint region has corners on the coordinate axes, the solution sets many coefficients exactly to zero, so the lasso performs feature selectionfeature selection, n.: Choosing which of the available input measurements a model should use and dropping the rest, so the model becomes simpler, faster and often more accurate. and shrinkage at once and gives sparse, interpretable models even when features outnumber observations. There is no closed form: solutions are computed by coordinate descent, as in the glmnet software, or by least-angle regression (LARS), which traces the whole solution path as λ varies. With groups of correlated features it tends to keep one arbitrarily, it selects at most n features when there are n observations, and it shrinks large coefficients too much — motivations for the elastic netelastic net, n.: A blend of two ways of keeping a model simple: it can drop useless inputs entirely while treating groups of similar inputs alike, instead of picking one of them at random. and adaptive or relaxed variants.

    The lasso kept 14 of the 600 candidate biomarkers, which made the model easy to validate in the lab.

β^lasso=arg⁡min⁡β12n∥y−Xβ∥22+λ∥β∥1\hat{\boldsymbol\beta}_{\text{lasso}} = \arg\min_{\boldsymbol\beta} \frac{1}{2n}\lVert \mathbf{y} - X\boldsymbol\beta \rVert_2^2 + \lambda \lVert \boldsymbol\beta \rVert_1

Origin

Named and popularized by Robert Tibshirani in 'Regression Shrinkage and Selection via the Lasso' (Journal of the Royal Statistical Society, Series B, 1996); the name is also read as an acronym, 'least absolute shrinkage and selection operator'. L1-penalized fitting had earlier appeared in geophysics (Fadil Santosa and William Symes, 1986), and the same problem became basis pursuit denoising in signal processing (Scott Chen, David Donoho and Michael Saunders, 1998).

Usage note

Tibshirani writes 'lasso' in lowercase. Lasso selections are unstable under resampling, so the chosen features should not be read as the true causes. The L1 idea also underlies compressed sensing and sparse coding, including sparse autoencoderssparse autoencoder, n.: A small add-on network trained to break an AI model's tangled internal signals into a much larger set of simpler pieces, most of which can be labeled with an understandable concept..

Related

See also
  • regularizationregularization, n.: Any technique that stops a model from memorizing its training examples too closely — often by penalizing complexity — so that it works better on new data.
  • ridge regressionridge regression, n.: Linear regression with a built-in brake: it fits the data while also keeping the weights small, which makes predictions more stable when there are many overlapping inputs.
  • elastic netelastic net, n.: A blend of two ways of keeping a model simple: it can drop useless inputs entirely while treating groups of similar inputs alike, instead of picking one of them at random.
  • feature selectionfeature selection, n.: Choosing which of the available input measurements a model should use and dropping the rest, so the model becomes simpler, faster and often more accurate.
  • linear regressionlinear regression, n.: Fitting a straight line (or flat surface) through data points to predict one number from others — say, a house's price from its floor area — choosing the line that keeps prediction errors smallest.
  • sparsitysparsity, n.: The property of being mostly zeros or mostly switched off. AI systems exploit it by skipping the empty parts, so they need less memory and computation.
Contrast with
  • ridge regressionridge regression, n.: Linear regression with a built-in brake: it fits the data while also keeping the weights small, which makes predictions more stable when there are many overlapping inputs.

Sources

  1. Tibshirani, R. (1996). Regression Shrinkage and Selection via the Lasso. Journal of the Royal Statistical Society, Series B, 58(1), 267–288.

Mentioned in

convex optimizationconvex optimization, n.: Finding the lowest point of a bowl-shaped landscape. Because a bowl has only one bottom, such problems can be solved reliably — unlike training a neural network, whose landscape is bumpy., feature selectionfeature selection, n.: Choosing which of the available input measurements a model should use and dropping the rest, so the model becomes simpler, faster and often more accurate., linear regressionlinear regression, n.: Fitting a straight line (or flat surface) through data points to predict one number from others — say, a house's price from its floor area — choosing the line that keeps prediction errors smallest., normnorm, n.: A way of measuring the size, or length, of a list of numbers — for example the straight-line distance from the origin to the point the list describes., regularizationregularization, n.: Any technique that stops a model from memorizing its training examples too closely — often by penalizing complexity — so that it works better on new data., ridge regressionridge regression, n.: Linear regression with a built-in brake: it fits the data while also keeping the weights small, which makes predictions more stable when there are many overlapping inputs., sparsitysparsity, n.: The property of being mostly zeros or mostly switched off. AI systems exploit it by skipping the empty parts, so they need less memory and computation., dictionary learningdictionary learning, n.: A method that learns a large vocabulary of basic patterns so that any piece of data can be described using just a few of them. In AI interpretability it builds a dictionary of a model's own concepts., elastic netelastic net, n.: A blend of two ways of keeping a model simple: it can drop useless inputs entirely while treating groups of similar inputs alike, instead of picking one of them at random., Jerome FriedmanJerome Friedman, prop. n.: An American statistician who invented gradient boosting, the method behind XGBoost and other leading tools for spreadsheet-style data, and co-wrote a standard machine-learning textbook., maximum a posteriori estimationmaximum a posteriori estimation, n.: Picking the single most believable setting of a model after combining the evidence with what was assumed beforehand. In practice it often amounts to ordinary training plus a penalty that keeps settings modest.