Skip to content
Last Dictionary
Plane
Mathematics & StatisticsMachine LearningDeep Learning

singular value decomposition

nounAlso: SVD, singular value, singular values, singular vectors

In plain wordsA way of rewriting any grid of numbers as three simple moves in turn — turn, stretch along perpendicular axes, turn again — which reveals the directions that matter most in the data.

  1. The factorization of any real m × n matrixmatrix, n.: A rectangular grid of numbers arranged in rows and columns. Most of what a neural network has learned is stored in large grids like this. A as UΣVᵀ, where U and V are orthogonal matrices whose columns (the left and right singular vectors) form orthonormal bases and Σ is diagonal with nonnegative entries, the singular values, in decreasing order. Every linear map thus splits into a rotation or reflection, an axis-aligned scaling and another rotation. The number of nonzero singular values is the matrix's rankrank, n.: The number of truly independent directions a grid of numbers contains. A low-rank grid can be rebuilt exactly from a few simple building blocks, however large it looks., the largest is its spectral normnorm, n.: A way of measuring the size, or length, of a list of numbers — for example the straight-line distance from the origin to the point the list describes., and the ratio of largest to smallest is its condition number, a gauge of numerical stabilitynumerical stability, n.: Whether a computer calculation stays accurate despite the tiny rounding errors in every step, instead of blowing up into nonsense such as infinities.. Truncating the SVD gives the best low-rank approximationlow-rank approximation, n.: Replacing a big grid of numbers with a slimmer one that keeps most of the same information, the way a few pencil strokes can capture a face. It saves memory and computation., the basis of principal component analysisprincipal component analysis, n.: A technique that finds the few directions in which data vary the most and describes each item by its position along them, compressing many measurements into a handful of summary scores., latent semantic analysislatent semantic analysis, n.: An early technique for capturing word meaning by counting which words appear in which documents, then compressing those counts so that words used in similar contexts end up close together. and many compression methods. In deep learning, SVD is used to study weight spectra, initialize and compress layers, and analyze the orthogonalized updates of optimizers such as MuonMuon, prop. n.: A newer training method for neural networks that tidies each update so it pushes evenly in all directions within a weight table. Several groups report that it trains language models with less computing than the standard method..

    The SVD of the fine-tuning update showed almost all its energy in the top eight singular values.

A=UΣV⊤=∑i=1rσi uivi⊤A = U\Sigma V^{\top} = \sum_{i=1}^{r} \sigma_i\,\mathbf{u}_i\mathbf{v}_i^{\top}

Origin

Discovered independently by Eugenio Beltrami (1873) and Camille Jordan (1874) and extended by James Joseph Sylvester (1889), Erhard Schmidt (1907) and Hermann Weyl (1912); 'singular value' in its modern sense was settled by the 1930s. Carl Eckart and Gale Young's 1936 approximation theorem made it central to data analysis, and Gene Golub and William Kahan's 1965 algorithm made it practical on computers.

Related

See also
  • eigenvectoreigenvector, n.: A special direction for a transformation: anything pointing that way stays on its line, only longer, shorter or reversed; the amount of stretch is the eigenvalue. Together they reveal the main axes along which data or a system varies.
  • low-rank approximationlow-rank approximation, n.: Replacing a big grid of numbers with a slimmer one that keeps most of the same information, the way a few pencil strokes can capture a face. It saves memory and computation.
  • principal component analysisprincipal component analysis, n.: A technique that finds the few directions in which data vary the most and describes each item by its position along them, compressing many measurements into a handful of summary scores.
  • rankrank, n.: The number of truly independent directions a grid of numbers contains. A low-rank grid can be rebuilt exactly from a few simple building blocks, however large it looks.
  • latent semantic analysislatent semantic analysis, n.: An early technique for capturing word meaning by counting which words appear in which documents, then compressing those counts so that words used in similar contexts end up close together.
  • low-rank adaptationlow-rank adaptation, n.: A cheap way to customize a big AI model: instead of changing all its billions of numbers, you train a small add-on of a few million numbers that slightly adjusts how it behaves.
  • matrixmatrix, n.: A rectangular grid of numbers arranged in rows and columns. Most of what a neural network has learned is stored in large grids like this.

Sources

  1. Golub, G. & Kahan, W. (1965). Calculating the Singular Values and Pseudo-Inverse of a Matrix. Journal of the Society for Industrial and Applied Mathematics, Series B: Numerical Analysis, 2(2).

Mentioned in

eigenvectoreigenvector, n.: A special direction for a transformation: anything pointing that way stays on its line, only longer, shorter or reversed; the amount of stretch is the eigenvalue. Together they reveal the main axes along which data or a system varies., linear algebralinear algebra, n.: The branch of mathematics about lists and grids of numbers and the even, straight-line transformations between them. It is the everyday language in which neural networks are written and computed., low-rank approximationlow-rank approximation, n.: Replacing a big grid of numbers with a slimmer one that keeps most of the same information, the way a few pencil strokes can capture a face. It saves memory and computation., matrixmatrix, n.: A rectangular grid of numbers arranged in rows and columns. Most of what a neural network has learned is stored in large grids like this., matrix factorizationmatrix factorization, n.: Breaking a big table of numbers, such as which users liked which films, into two smaller tables of hidden traits, so missing entries can be filled in by multiplying them back together., MuonMuon, prop. n.: A newer training method for neural networks that tidies each update so it pushes evenly in all directions within a weight table. Several groups report that it trains language models with less computing than the standard method., ordinary least squaresordinary least squares, n.: The standard way to fit a line through data: choose the line that makes the total of the squared vertical distances from the points to the line as small as possible., principal component analysisprincipal component analysis, n.: A technique that finds the few directions in which data vary the most and describes each item by its position along them, compressing many measurements into a handful of summary scores., rankrank, n.: The number of truly independent directions a grid of numbers contains. A low-rank grid can be rebuilt exactly from a few simple building blocks, however large it looks., latent semantic analysislatent semantic analysis, n.: An early technique for capturing word meaning by counting which words appear in which documents, then compressing those counts so that words used in similar contexts end up close together., orthogonalityorthogonality, n.: The mathematical version of being at right angles. Two directions are orthogonal when moving along one tells you nothing about the other, like north and east on a map.