Skip to content
Last Dictionary
Plane
Language Models & TransformersDeep Learning

multi-head attention

nounAlso: MHA, multi-headed attention

In plain wordsRunning several attention processes side by side, each free to track a different kind of relationship between words — grammar, topic, who did what — and then combining their results.

  1. An attentionattention, n.: A way for an AI model to choose what to focus on. For each word it processes, the model scores how relevant every other word is and draws information mostly from the relevant ones. layer that divides its work among h smaller attention operations (headsattention head, n.: One of several parallel attention units inside a transformer layer. Each head learns its own way of deciding which words to connect, so different heads can specialize in different patterns.) running side by side, so that a single layer can follow several kinds of relationship between positions at once. Each head views the input through its own learned query, key and value matrices, which shrink the model width d_model to a head dimension of typically d_model/h; the heads' results are joined end to end and mixed by one more learned matrix. Because each head is narrow, the whole layer costs about as much as one attention operation at full width. The 2017 transformertransformer, n.: The design behind today's chatbots and much of modern AI. It handles text by letting every word look at every other word and decide which of them matter for understanding it. used 8 heads of dimension 64 in a 512-dimensional model; large language models use dozens of heads per layer. Because every head keeps its own keys and values, the KV cacheKV cache, n.: A memory store where a chatbot keeps its working notes on everything already read and written, so each new word does not require re-reading the whole conversation from scratch. grows with the number of heads, which motivated multi-query attentionmulti-query attention, n.: A cheaper form of attention in which all the attention heads share one store of information about earlier words instead of each keeping its own, making text generation faster and lighter on memory., grouped-query attentiongrouped-query attention, n.: A middle ground between full multi-head attention and its cheapest variant: attention heads are split into groups, and each group shares one store of information about earlier words, saving memory with little loss in quality. and multi-head latent attentionmulti-head latent attention, n.: An attention design from the Chinese lab DeepSeek that compresses the stored information about earlier words into a much smaller summary, so long conversations need far less memory..

    Each layer's multi-head attention uses 32 heads of dimension 128, matching the 4,096-wide model.

MultiHead(X)=Concat(head1,…,headh) WO,headi=Attention(XWiQ, XWiK, XWiV)\mathrm{MultiHead}(X) = \mathrm{Concat}(\mathrm{head}_1, \dots, \mathrm{head}_h)\,W^{O}, \quad \mathrm{head}_i = \mathrm{Attention}(XW_i^{Q},\, XW_i^{K},\, XW_i^{V})

Origin

Introduced in 'Attention Is All You Need' (Vaswani et al., 2017), whose ablations on English–German translation found single-head attention about 0.9 BLEU worse than the best multi-head setting, with quality also falling when heads became too many.

Usage note

As an abbreviation, MHA usually denotes the original design in which every head has its own keys and values, as opposed to MQA, GQA and MLA.

Related

See also
  • attention headattention head, n.: One of several parallel attention units inside a transformer layer. Each head learns its own way of deciding which words to connect, so different heads can specialize in different patterns.
  • self-attentionself-attention, n.: Attention in which a passage looks at itself: every word checks every other word in the same text to work out its meaning in context, such as which noun 'it' refers to.
  • scaled dot-product attentionscaled dot-product attention, n.: The standard formula for attention in transformers: score how well each question matches each label, scale the scores down, turn them into percentages, and use those to blend the stored information.
  • grouped-query attentiongrouped-query attention, n.: A middle ground between full multi-head attention and its cheapest variant: attention heads are split into groups, and each group shares one store of information about earlier words, saving memory with little loss in quality.
  • multi-query attentionmulti-query attention, n.: A cheaper form of attention in which all the attention heads share one store of information about earlier words instead of each keeping its own, making text generation faster and lighter on memory.
  • multi-head latent attentionmulti-head latent attention, n.: An attention design from the Chinese lab DeepSeek that compresses the stored information about earlier words into a much smaller summary, so long conversations need far less memory.
  • transformertransformer, n.: The design behind today's chatbots and much of modern AI. It handles text by letting every word look at every other word and decide which of them matter for understanding it.

Sources

  1. Vaswani, A. et al. (2017). Attention Is All You Need. NIPS 2017. arxiv.org/abs/1706.03762

Mentioned in

attentionattention, n.: A way for an AI model to choose what to focus on. For each word it processes, the model scores how relevant every other word is and draws information mostly from the relevant ones., transformertransformer, n.: The design behind today's chatbots and much of modern AI. It handles text by letting every word look at every other word and decide which of them matter for understanding it., attention headattention head, n.: One of several parallel attention units inside a transformer layer. Each head learns its own way of deciding which words to connect, so different heads can specialize in different patterns., Noam ShazeerNoam Shazeer, prop. n.: An American computer scientist who co-invented the transformer and several techniques found in nearly every large language model, co-founded Character.AI, and later returned to Google to co-lead Gemini., scaled dot-product attentionscaled dot-product attention, n.: The standard formula for attention in transformers: score how well each question matches each label, scale the scores down, turn them into percentages, and use those to blend the stored information., self-attentionself-attention, n.: Attention in which a passage looks at itself: every word checks every other word in the same text to work out its meaning in context, such as which noun 'it' refers to., transformer blocktransformer block, n.: The repeating building unit of a transformer. A model is a stack of dozens of identical blocks, each pairing an attention step that mixes information between words with a step that processes each word., Ashish VaswaniAshish Vaswani, prop. n.: An Indian-born computer scientist listed first on 'Attention Is All You Need', the 2017 paper that introduced the transformer, the design behind today's language models., grouped-query attentiongrouped-query attention, n.: A middle ground between full multi-head attention and its cheapest variant: attention heads are split into groups, and each group shares one store of information about earlier words, saving memory with little loss in quality., multi-head latent attentionmulti-head latent attention, n.: An attention design from the Chinese lab DeepSeek that compresses the stored information about earlier words into a much smaller summary, so long conversations need far less memory., multi-query attentionmulti-query attention, n.: A cheaper form of attention in which all the attention heads share one store of information about earlier words instead of each keeping its own, making text generation faster and lighter on memory.