Skip to content
Last Dictionary
Plane
Deep LearningLanguage Models & Transformers

RMSNorm

nounAlso: root mean square layer normalization, RMS norm, RMS normalization

In plain wordsA simpler, faster version of the layer that keeps a network's internal numbers at a steady scale. Most recent large language models use it.

  1. A normalization layernormalization layer, n.: A layer that keeps the numbers passing through a neural network within a steady range by shifting and resizing them, so that deep networks learn faster and are less likely to break down during training. that divides each example's activation vector by its root mean square and multiplies by a learned per-feature gain, omitting the mean subtraction (re-centering) and usually the bias of layer normalizationlayer normalization, n.: A layer that rescales the numbers inside a network, separately for each example, so they keep a steady average and spread. It, or a simpler variant, is a standard part of the models behind today's chatbots.. Its authors argued that re-scaling invariance, not re-centering, is what makes LayerNorm effective; dropping the mean computation saves time, and they reported running-time reductions of 7–64% across models at comparable accuracy. As of 2026 it is the normalization used by LLaMA, Mistral, Qwen, DeepSeek and most other open-weight language modelslarge language model, n.: A computer program trained on enormous amounts of text to predict the words that come next. At vast scale, that simple skill lets it chat, answer questions, translate, summarize and write code., typically in a pre-norm arrangement; QK-normQK-norm, n.: A small change to how a language model pays attention — rescaling two sets of internal numbers before comparing them — that prevents a common way large models' training goes haywire. applies such a normalization (in many recent models, RMSNorm) to attention queries and keys to prevent instabilitiestraining instability, n.: The tendency of very large training runs to go wrong — the error suddenly jumping or blowing up — which can waste weeks of expensive computing unless engineers prevent or repair it. from growing attention logits.

    The model uses RMSNorm with epsilon 1e-6 before every attention and feed-forward block.

RMSNorm⁡(x)=x1d∑i=1dxi2+ϵ⊙g\operatorname{RMSNorm}(\mathbf{x}) = \frac{\mathbf{x}}{\sqrt{\frac{1}{d}\sum_{i=1}^{d} x_i^2 + \epsilon}} \odot \mathbf{g}

Origin

Introduced by Biao Zhang and Rico Sennrich of the University of Edinburgh in 'Root Mean Square Layer Normalization' (arXiv, October 2019; NeurIPS 2019). Google's T5 (2019) used a similar scale-only layer normalization, and Meta's LLaMA (2023) made RMSNorm the default for open-weight models.

Related

See also
  • layer normalizationlayer normalization, n.: A layer that rescales the numbers inside a network, separately for each example, so they keep a steady average and spread. It, or a simpler variant, is a standard part of the models behind today's chatbots.
  • normalization layernormalization layer, n.: A layer that keeps the numbers passing through a neural network within a steady range by shifting and resizing them, so that deep networks learn faster and are less likely to break down during training.
  • batch normalizationbatch normalization, n.: A layer that rescales the numbers flowing through a network so that, across each batch of training examples, they keep a steady average and spread. It makes deep networks train faster and more reliably.
  • transformertransformer, n.: The design behind today's chatbots and much of modern AI. It handles text by letting every word look at every other word and decide which of them matter for understanding it.
  • large language modellarge language model, n.: A computer program trained on enormous amounts of text to predict the words that come next. At vast scale, that simple skill lets it chat, answer questions, translate, summarize and write code.

Sources

  1. Zhang, B. & Sennrich, R. (2019). Root Mean Square Layer Normalization. arXiv:1910.07467; NeurIPS 2019. arxiv.org/abs/1910.07467

Mentioned in

batch normalizationbatch normalization, n.: A layer that rescales the numbers flowing through a network so that, across each batch of training examples, they keep a steady average and spread. It makes deep networks train faster and more reliably., layer normalizationlayer normalization, n.: A layer that rescales the numbers inside a network, separately for each example, so they keep a steady average and spread. It, or a simpler variant, is a standard part of the models behind today's chatbots., normnorm, n.: A way of measuring the size, or length, of a list of numbers — for example the straight-line distance from the origin to the point the list describes., normalization layernormalization layer, n.: A layer that keeps the numbers passing through a neural network within a steady range by shifting and resizing them, so that deep networks learn faster and are less likely to break down during training., training instabilitytraining instability, n.: The tendency of very large training runs to go wrong — the error suddenly jumping or blowing up — which can waste weeks of expensive computing unless engineers prevent or repair it., transformer blocktransformer block, n.: The repeating building unit of a transformer. A model is a stack of dozens of identical blocks, each pairing an attention step that mixes information between words with a step that processes each word., QK-normQK-norm, n.: A small change to how a language model pays attention — rescaling two sets of internal numbers before comparing them — that prevents a common way large models' training goes haywire.