query, key and value
In plain wordsThe three roles each word plays inside attention: the query is what it is looking for, the key is what it can be found by, and the value is the information it hands over when chosen.
The three vectors computed from each input position by separate learned linear projections in an attentionattention, n.: A way for an AI model to choose what to focus on. For each word it processes, the model scores how relevant every other word is and draws information mostly from the relevant ones. layer. A query represents what a position is seeking; keys are compared with the query (by dot productscaled dot-product attention, n.: The standard formula for attention in transformers: score how well each question matches each label, scale the scores down, turn them into percentages, and use those to blend the stored information. in transformers) to produce attention weights; values are the content actually mixed into the output according to those weights. Separating keys from values lets the features used for matching differ from the information transmitted. In self-attentionself-attention, n.: Attention in which a passage looks at itself: every word checks every other word in the same text to work out its meaning in context, such as which noun 'it' refers to. all three come from the same sequence; in cross-attentioncross-attention, n.: Attention between two different inputs: one sequence consults another, as when a translation model writing French looks back at the English sentence, or an image generator consults the text prompt. the query comes from one sequence and keys and values from another. During generation, the keys and values of earlier tokens are stored in the KV cacheKV cache, n.: A memory store where a chatbot keeps its working notes on everything already read and written, so each new word does not require re-reading the whole conversation from scratch. and only one new query is computed per step. Interpretability work analyzes the query–key and value–output products separately as QK and OV circuitsQK and OV circuits, n.: A way of splitting what each attention unit in a language model does into two separate questions: where should it look, and what should it pass along from what it sees..
The key and value projections are shared across groups of heads, but each head keeps its own query projection.
Origin
The vocabulary comes from information retrieval and associative memory, where a query is matched against keys to retrieve stored values. Key–value memory networks (Alexander Miller and colleagues, 2016) used the split before the transformer; 'Attention Is All You Need' (2017) fixed the query–key–value formulation now standard.
Usage note
The database analogy is loose: attention returns a soft, weighted blend of all values rather than one exact match. Q, K and V usually denote matrices stacking the vectors for all positions.
Related
- attentionattention, n.: A way for an AI model to choose what to focus on. For each word it processes, the model scores how relevant every other word is and draws information mostly from the relevant ones.
- scaled dot-product attentionscaled dot-product attention, n.: The standard formula for attention in transformers: score how well each question matches each label, scale the scores down, turn them into percentages, and use those to blend the stored information.
- self-attentionself-attention, n.: Attention in which a passage looks at itself: every word checks every other word in the same text to work out its meaning in context, such as which noun 'it' refers to.
- KV cacheKV cache, n.: A memory store where a chatbot keeps its working notes on everything already read and written, so each new word does not require re-reading the whole conversation from scratch.
- QK and OV circuitsQK and OV circuits, n.: A way of splitting what each attention unit in a language model does into two separate questions: where should it look, and what should it pass along from what it sees.
- attention headattention head, n.: One of several parallel attention units inside a transformer layer. Each head learns its own way of deciding which words to connect, so different heads can specialize in different patterns.
Sources
- Vaswani, A. et al. (2017). Attention Is All You Need. NIPS 2017. arxiv.org/abs/1706.03762
Mentioned in
attentionattention, n.: A way for an AI model to choose what to focus on. For each word it processes, the model scores how relevant every other word is and draws information mostly from the relevant ones., KV cacheKV cache, n.: A memory store where a chatbot keeps its working notes on everything already read and written, so each new word does not require re-reading the whole conversation from scratch., scaled dot-product attentionscaled dot-product attention, n.: The standard formula for attention in transformers: score how well each question matches each label, scale the scores down, turn them into percentages, and use those to blend the stored information., self-attentionself-attention, n.: Attention in which a passage looks at itself: every word checks every other word in the same text to work out its meaning in context, such as which noun 'it' refers to., prefix tuningprefix tuning, n.: Customizing a frozen language model by learning a short sequence of invisible 'virtual words' that is placed in front of every input at every layer, steering what the model produces., QK and OV circuitsQK and OV circuits, n.: A way of splitting what each attention unit in a language model does into two separate questions: where should it look, and what should it pass along from what it sees., QK-normQK-norm, n.: A small change to how a language model pays attention — rescaling two sets of internal numbers before comparing them — that prevents a common way large models' training goes haywire., rotary positional embeddingrotary positional embedding, n.: A way of telling a transformer where each word is by rotating parts of its internal numbers by an angle that grows with position, so the model can read off how far apart two words are.