Skip to content
Last Dictionary
Plane
Mathematics & StatisticsDeep LearningMachine Learning

norm

nounAlso: vector norm, L1 norm, L2 norm, Euclidean norm, Frobenius norm, spectral norm, magnitude

In plain wordsA way of measuring the size, or length, of a list of numbers — for example the straight-line distance from the origin to the point the list describes.

  1. A function that assigns a nonnegative length to each vectorvector, n.: An ordered list of numbers, like map coordinates. AI systems turn words, images and users into long lists of numbers so that similar things end up as nearby points., is zero only for the zero vector, scales with scalar multiplication (‖cx‖ = |c|‖x‖) and obeys the triangle inequality (‖x + y‖ ≤ ‖x‖ + ‖y‖). The most common are the Lp norms: L2, the Euclidean length (square root of the sum of squares); L1, the sum of absolute values; and L∞, the largest absolute component. Matrices have norms too, notably the Frobenius norm (L2 over all entries) and the spectral norm (the largest singular value). Norms are everywhere in machine learning: L2 and L1 penalties underlie weight decayweight decay, n.: A training rule that shrinks every weight slightly at each step, like a small tax on large numbers. It keeps models simpler and usually helps them work better on new data., ridge regressionridge regression, n.: Linear regression with a built-in brake: it fits the data while also keeping the weights small, which makes predictions more stable when there are many overlapping inputs. and the lassolasso, n.: A version of linear regression that keeps only the most useful inputs: it penalizes the total size of the weights so that many become exactly zero, leaving a simpler model.; gradient clippinggradient clipping, n.: A safety limit on how big a single training update can be. If the computed adjustment is unusually large, it is scaled down so that one bad batch cannot wreck the model. rescales gradients whose norm exceeds a threshold; layer normalizationlayer normalization, n.: A layer that rescales the numbers inside a network, separately for each example, so they keep a steady average and spread. It, or a simpler variant, is a standard part of the models behind today's chatbots. and RMSNormRMSNorm, n.: A simpler, faster version of the layer that keeps a network's internal numbers at a steady scale. Most recent large language models use it. rescale activations; and the distance between two vectors is the norm of their difference.

    Clip the global gradient norm to 1.0 before each optimizer step.

∥x∥p=(∑i∣xi∣p)1/p\lVert\mathbf{x}\rVert_p = \Big(\sum_{i} |x_i|^p\Big)^{1/p}

Origin

Carl Friedrich Gauss used Latin norma in 1832 for a² + b², the norm of a complex integer. Erhard Schmidt defined the Euclidean norm of vectors in infinite-dimensional space in 1908, and Stefan Banach axiomatized normed spaces in his dissertation, published in 1922.

Usage note

Normalization layers are related but not identical: layer normalizationlayer normalization, n.: A layer that rescales the numbers inside a network, separately for each example, so they keep a steady average and spread. It, or a simpler variant, is a standard part of the models behind today's chatbots. subtracts the mean and divides by the standard deviation, while RMSNormRMSNorm, n.: A simpler, faster version of the layer that keeps a network's internal numbers at a steady scale. Most recent large language models use it. divides by the root mean square (the L2 norm over √d). The 'normal' of normal distributionnormal distribution, n.: The familiar bell-shaped curve: most values cluster near the average, and values become rapidly rarer the farther they lie from it. Measurement errors and adult heights roughly follow it. is an unrelated sense.

Related

See also
  • vectorvector, n.: An ordered list of numbers, like map coordinates. AI systems turn words, images and users into long lists of numbers so that similar things end up as nearby points.
  • Euclidean distanceEuclidean distance, n.: How far apart two points are in a straight line, as a ruler would measure it, extended from two or three dimensions to lists of numbers of any length.
  • cosine similaritycosine similarity, n.: A score from −1 to 1 for how closely two lists of numbers point the same way, ignoring their size. It is the standard test of whether two texts mean similar things.
  • regularizationregularization, n.: Any technique that stops a model from memorizing its training examples too closely — often by penalizing complexity — so that it works better on new data.
  • gradient clippinggradient clipping, n.: A safety limit on how big a single training update can be. If the computed adjustment is unusually large, it is scaled down so that one bad batch cannot wreck the model.
  • layer normalizationlayer normalization, n.: A layer that rescales the numbers inside a network, separately for each example, so they keep a steady average and spread. It, or a simpler variant, is a standard part of the models behind today's chatbots.
  • weight decayweight decay, n.: A training rule that shrinks every weight slightly at each step, like a small tax on large numbers. It keeps models simpler and usually helps them work better on new data.

Mentioned in

vectorvector, n.: An ordered list of numbers, like map coordinates. AI systems turn words, images and users into long lists of numbers so that similar things end up as nearby points., cosine similaritycosine similarity, n.: A score from −1 to 1 for how closely two lists of numbers point the same way, ignoring their size. It is the standard test of whether two texts mean similar things., dot productdot product, n.: A score of how well two lists of numbers line up, found by multiplying matching entries and adding the products. It is large when the lists point the same way and zero when they are at right angles., Euclidean distanceEuclidean distance, n.: How far apart two points are in a straight line, as a ruler would measure it, extended from two or three dimensions to lists of numbers of any length., feature scalingfeature scaling, n.: Putting input measurements on comparable scales, so that income in dollars and age in years, say, both span similar ranges and no feature dominates just because of its units., low-rank approximationlow-rank approximation, n.: Replacing a big grid of numbers with a slimmer one that keeps most of the same information, the way a few pencil strokes can capture a face. It saves memory and computation., normal distributionnormal distribution, n.: The familiar bell-shaped curve: most values cluster near the average, and values become rapidly rarer the farther they lie from it. Measurement errors and adult heights roughly follow it., singular value decompositionsingular value decomposition, n.: A way of rewriting any grid of numbers as three simple moves in turn — turn, stretch along perpendicular axes, turn again — which reveals the directions that matter most in the data.