sparse attention
In plain wordsAttention in which each word looks at only a selected subset of the other words instead of all of them, so very long texts can be processed at a fraction of the cost.
Any attentionattention, n.: A way for an AI model to choose what to focus on. For each word it processes, the model scores how relevant every other word is and draws information mostly from the relevant ones. scheme in which each query attends to a subset of key positions rather than all of them, reducing the quadratic cost of full self-attentionself-attention, n.: Attention in which a passage looks at itself: every word checks every other word in the same text to work out its meaning in context, such as which noun 'it' refers to.. Fixed patterns, set in advance, include local windows (sliding window attentionsliding window attention, n.: Attention in which each word looks only at a fixed number of nearby earlier words rather than the whole text, keeping the cost of very long inputs under control.), strided or dilated positions, blocks, and a few global tokens that see everything (Sparse Transformer, Longformer, BigBird). Content-based patterns choose keys per query, for example by hashing similar vectors into buckets (Reformer) or by top-k selection over compressed blocks. Recent designs make the selection trainable and hardware-aligned: DeepSeek's Native Sparse Attention (early 2025) combines compressed, selected and sliding-window branches, and DeepSeek-V3.2-Exp (September 2025) added a lightweight indexer that picks which past tokens each query attends to. Real speedups depend on GPU kernels that skip blocks efficiently.
With block-sparse attention, prefill at 128K tokens ran three times faster.
Origin
Rewon Child, Scott Gray, Alec Radford and Ilya Sutskever of OpenAI introduced factorized sparse attention in 'Generating Long Sequences with Sparse Transformers' (April 2019), reducing cost to O(n√n); Longformer, BigBird and Reformer followed in 2020.
Usage note
Distinct from sparsitysparsity, n.: The property of being mostly zeros or mostly switched off. AI systems exploit it by skipping the empty parts, so they need less memory and computation. of weights or activations, and from the sparse activation of parameters in a mixture of expertsmixture of experts, n.: A way to build very large AI models cheaply: the model contains many specialist sub-networks, and for each word only a few are switched on, picked by a small dispatcher network..
Related
- sliding window attentionsliding window attention, n.: Attention in which each word looks only at a fixed number of nearby earlier words rather than the whole text, keeping the cost of very long inputs under control.
- linear attentionlinear attention, n.: A family of faster substitutes for attention whose cost grows in step with the length of the text rather than with its square, by keeping a running summary instead of comparing every pair of words.
- long contextlong context, n.: The ability of an AI model to work with very long inputs — whole books, codebases or hours of transcripts — in one go, and the techniques that make this possible.
- self-attentionself-attention, n.: Attention in which a passage looks at itself: every word checks every other word in the same text to work out its meaning in context, such as which noun 'it' refers to.
- sparsitysparsity, n.: The property of being mostly zeros or mostly switched off. AI systems exploit it by skipping the empty parts, so they need less memory and computation.
- FlashAttentionFlashAttention, prop. n.: A faster way for AI chips to compute attention, the step in which a language model relates every word to every other. It gets exactly the same answer while moving far less data around the chip.
- sparsitysparsity, n.: The property of being mostly zeros or mostly switched off. AI systems exploit it by skipping the empty parts, so they need less memory and computation.
- mixture of expertsmixture of experts, n.: A way to build very large AI models cheaply: the model contains many specialist sub-networks, and for each word only a few are switched on, picked by a small dispatcher network.
Sources
- Child, R., Gray, S., Radford, A. & Sutskever, I. (2019). Generating Long Sequences with Sparse Transformers. arXiv:1904.10509. arxiv.org/abs/1904.10509
Mentioned in
attentionattention, n.: A way for an AI model to choose what to focus on. For each word it processes, the model scores how relevant every other word is and draws information mostly from the relevant ones., transformertransformer, n.: The design behind today's chatbots and much of modern AI. It handles text by letting every word look at every other word and decide which of them matter for understanding it., attention maskattention mask, n.: A grid of yes-or-no switches telling an attention layer which words each word may look at — for example, ignore blank padding, or do not peek at words that come later., big O notationbig O notation, n.: Shorthand for how the cost of a computation grows as the problem gets bigger. Saying attention costs 'O(n²)' means doubling the length of the text roughly quadruples the work., FlashAttentionFlashAttention, prop. n.: A faster way for AI chips to compute attention, the step in which a language model relates every word to every other. It gets exactly the same answer while moving far less data around the chip., long contextlong context, n.: The ability of an AI model to work with very long inputs — whole books, codebases or hours of transcripts — in one go, and the techniques that make this possible., mixture of expertsmixture of experts, n.: A way to build very large AI models cheaply: the model contains many specialist sub-networks, and for each word only a few are switched on, picked by a small dispatcher network., sparsitysparsity, n.: The property of being mostly zeros or mostly switched off. AI systems exploit it by skipping the empty parts, so they need less memory and computation., linear attentionlinear attention, n.: A family of faster substitutes for attention whose cost grows in step with the length of the text rather than with its square, by keeping a running summary instead of comparing every pair of words., sliding window attentionsliding window attention, n.: Attention in which each word looks only at a fixed number of nearby earlier words rather than the whole text, keeping the cost of very long inputs under control.