Skip to content
Last Dictionary
Plane
Language Models & TransformersDeep LearningPhilosophy & Mind

attention

nounAlso: attention mechanism, neural attention, soft attention

In plain wordsA way for an AI model to choose what to focus on. For each word it processes, the model scores how relevant every other word is and draws information mostly from the relevant ones.

  1. 1

    Neural networks

    A mechanism that lets a neural network compute each output as a weighted average of a set of value vectors, with weights computed from the data itself: a queryquery, key and value, n.: The three roles each word plays inside attention: the query is what it is looking for, the key is what it can be found by, and the value is the information it hands over when chosen. is scored against every key, the scores are normalized (usually with softmaxsoftmax, n.: A formula that turns a list of raw scores into probabilities — all positive and adding up to one — with the biggest share going to the highest score.) and the resulting weights mix the corresponding values. Because the weights are recomputed for every input, the network can route information from whichever positions are relevant, however far away. Variants differ in the scoring function (additive or dot-productscaled dot-product attention, n.: The standard formula for attention in transformers: score how well each question matches each label, scale the scores down, turn them into percentages, and use those to blend the stored information.), in what attends to what (self-attentionself-attention, n.: Attention in which a passage looks at itself: every word checks every other word in the same text to work out its meaning in context, such as which noun 'it' refers to., cross-attentioncross-attention, n.: Attention between two different inputs: one sequence consults another, as when a translation model writing French looks back at the English sentence, or an image generator consults the text prompt.), in which positions are visible (causalcausal mask, n.: A rule inside text-generating AI that stops each word from seeing the words after it, so the model learns to predict what comes next without being able to cheat., windowedsliding window attention, n.: Attention in which each word looks only at a fixed number of nearby earlier words rather than the whole text, keeping the cost of very long inputs under control., sparsesparse attention, n.: Attention in which each word looks at only a selected subset of the other words instead of all of them, so very long texts can be processed at a fraction of the cost.) and in how keys and values are shared across heads (MQAmulti-query attention, n.: A cheaper form of attention in which all the attention heads share one store of information about earlier words instead of each keeping its own, making text generation faster and lighter on memory., GQAgrouped-query attention, n.: A middle ground between full multi-head attention and its cheapest variant: attention heads are split into groups, and each group shares one store of information about earlier words, saving memory with little loss in quality., MLAmulti-head latent attention, n.: An attention design from the Chinese lab DeepSeek that compresses the stored information about earlier words into a much smaller summary, so long conversations need far less memory.). It is the core operation of the transformertransformer, n.: The design behind today's chatbots and much of modern AI. It handles text by letting every word look at every other word and decide which of them matter for understanding it..

    Attention lets the decoder look straight at the relevant source words instead of squeezing the sentence into one vector.

  2. 2

    Psychology

    The mind's prioritizing of some sensations, places or thoughts for fuller processing while others are filtered out or neglected, a necessity because perception and thought have limited capacity; it has been studied experimentally since the late nineteenth century. Neural-network attention borrowed the word and the intuition of focusing on what matters, but it is a learned weighting operation, not a model of human attention; how far the two correspond is an open question in cognitive sciencecognitive science, n.: The scientific study of how minds work — how we reason, use language, remember and see — combining psychology, neuroscience, linguistics, philosophy, anthropology and artificial intelligence..

outi=∑jαij vj,αij=exp⁡s(qi,kj)∑j′exp⁡s(qi,kj′)\mathrm{out}_i = \sum_j \alpha_{ij}\, v_j, \qquad \alpha_{ij} = \frac{\exp s(q_i, k_j)}{\sum_{j'} \exp s(q_i, k_{j'})}

Origin

The neural-network sense took its modern form in Dzmitry Bahdanau, Kyunghyun Cho and Yoshua Bengio's 2014 machine-translation paper (ICLR 2015), whose decoder learned soft alignments to source words, 'a mechanism of attention' in their words. Neural models of visual attention go back to the 1980s; Minh-Thang Luong and colleagues (2015) simplified the scoring, and the 2017 transformertransformer, n.: The design behind today's chatbots and much of modern AI. It handles text by letting every word look at every other word and decide which of them matter for understanding it. made attention the network's central operation.

Usage note

In transformer writing, 'attention' usually means self-attentionself-attention, n.: Attention in which a passage looks at itself: every word checks every other word in the same text to work out its meaning in context, such as which noun 'it' refers to. unless stated otherwise. Attention weights show where information flows but are not by themselves a faithful explanation of a decision; see attention-based explanationattention-based explanation, n.: Explaining an AI model's output by showing which words or image patches its attention mechanism weighted most — an intuitive but disputed guide to what actually drove the output. and attention patternattention pattern, n.: A grid showing, for each word a language model processes, how much it is looking back at each earlier word at that moment — often drawn as a heat map..

Related

See also
  • self-attentionself-attention, n.: Attention in which a passage looks at itself: every word checks every other word in the same text to work out its meaning in context, such as which noun 'it' refers to.
  • cross-attentioncross-attention, n.: Attention between two different inputs: one sequence consults another, as when a translation model writing French looks back at the English sentence, or an image generator consults the text prompt.
  • scaled dot-product attentionscaled dot-product attention, n.: The standard formula for attention in transformers: score how well each question matches each label, scale the scores down, turn them into percentages, and use those to blend the stored information.
  • query, key and valuequery, key and value, n.: The three roles each word plays inside attention: the query is what it is looking for, the key is what it can be found by, and the value is the information it hands over when chosen.
  • multi-head attentionmulti-head attention, n.: Running several attention processes side by side, each free to track a different kind of relationship between words — grammar, topic, who did what — and then combining their results.
  • attention headattention head, n.: One of several parallel attention units inside a transformer layer. Each head learns its own way of deciding which words to connect, so different heads can specialize in different patterns.
  • transformertransformer, n.: The design behind today's chatbots and much of modern AI. It handles text by letting every word look at every other word and decide which of them matter for understanding it.
  • attention schema theoryattention schema theory, n.: A theory that the brain keeps a simplified sketch of its own attention, and that our feeling of having an inner awareness is simply how that sketch portrays attention.
Contrast with
  • recurrent neural networkrecurrent neural network, n.: A neural network that reads a sequence, such as a sentence or an audio clip, one piece at a time, carrying a running summary forward so that what came earlier can shape how later pieces are understood.

Sources

  1. Bahdanau, D., Cho, K. & Bengio, Y. (2014). Neural Machine Translation by Jointly Learning to Align and Translate. arXiv:1409.0473; ICLR 2015. arxiv.org/abs/1409.0473
  2. Luong, M.-T., Pham, H. & Manning, C. D. (2015). Effective Approaches to Attention-based Neural Machine Translation. EMNLP 2015. arxiv.org/abs/1508.04025

Mentioned in

layerlayer, n.: One stage in a neural network: a group of artificial neurons that all take the same input, transform it, and pass the result on to the next stage., transformertransformer, n.: The design behind today's chatbots and much of modern AI. It handles text by letting every word look at every other word and decide which of them matter for understanding it., attention headattention head, n.: One of several parallel attention units inside a transformer layer. Each head learns its own way of deciding which words to connect, so different heads can specialize in different patterns., Attention Is All You NeedAttention Is All You Need, prop. n.: The 2017 Google research paper that introduced the transformer, the neural-network design behind ChatGPT, Claude, Gemini and nearly every other modern AI language system., attention maskattention mask, n.: A grid of yes-or-no switches telling an attention layer which words each word may look at — for example, ignore blank padding, or do not peek at words that come later., continuous batchingcontinuous batching, n.: A way for an AI server to handle many conversations together, letting new ones join and finished ones leave after every word, instead of waiting for a whole group to finish., cross-attentioncross-attention, n.: Attention between two different inputs: one sequence consults another, as when a translation model writing French looks back at the English sentence, or an image generator consults the text prompt., encoder–decoder architectureencoder–decoder architecture, n.: A two-part model design: one part, the encoder, reads the input and condenses it into an internal summary; the other, the decoder, uses that summary to produce the output., feed-forward networkfeed-forward network, n.: The part of each transformer layer that works on every word separately, after attention has mixed in context. It holds most of a model's adjustable numbers and is thought to store much of its knowledge., graph neural networkgraph neural network, n.: A neural network for data shaped like a web of connections, such as molecules, social networks or road maps, in which each point repeatedly gathers information from its neighbors., Hopfield networkHopfield network, n.: A network of on–off units, all connected to one another, that stores patterns as stable states. Given a partial or noisy pattern, it settles into the closest stored one, like recalling a memory from a hint., ICLRICLR, prop. n.: A major deep-learning conference, held every spring since 2013, where the reviews of each submitted paper are posted online for anyone to read., image captioningimage captioning, n.: Having a computer write a short description of a picture, such as 'a dog catching a frisbee in a park', to help blind people, search engines and other AI systems., John JumperJohn Jumper, prop. n.: An American scientist who led Google DeepMind's AlphaFold project, which predicts the 3D shapes of proteins, and shared the 2024 Nobel Prize in Chemistry., Jürgen SchmidhuberJürgen Schmidhuber, prop. n.: A German computer scientist who co-invented the LSTM, a neural network with memory that powered much of speech recognition and translation in the 2010s, and who argues that early pioneers go under-credited., KV cacheKV cache, n.: A memory store where a chatbot keeps its working notes on everything already read and written, so each new word does not require re-reading the whole conversation from scratch., linear algebralinear algebra, n.: The branch of mathematics about lists and grids of numbers and the even, straight-line transformations between them. It is the everyday language in which neural networks are written and computed., machine translationmachine translation, n.: Software that renders writing or speech in one language into another language, as online translation services and phone translation apps do., MilaMila, prop. n.: Montreal's deep-learning hub, where professors and students from several Quebec universities work side by side. It grew from the lab Yoshua Bengio started in 1993 and is among the world's largest academic AI centers., multi-head attentionmulti-head attention, n.: Running several attention processes side by side, each free to track a different kind of relationship between words — grammar, topic, who did what — and then combining their results., prefillprefill, n.: The first stage of an AI's reply, in which it reads your whole prompt in one go before it starts writing. Long prompts make this stage take longer., query, key and valuequery, key and value, n.: The three roles each word plays inside attention: the query is what it is looking for, the key is what it can be found by, and the value is the information it hands over when chosen., rankrank, n.: The number of truly independent directions a grid of numbers contains. A low-rank grid can be rebuilt exactly from a few simple building blocks, however large it looks., scaled dot-product attentionscaled dot-product attention, n.: The standard formula for attention in transformers: score how well each question matches each label, scale the scores down, turn them into percentages, and use those to blend the stored information.