KV cache compression
In plain wordsTechniques that shrink the memory an AI model uses to remember a long conversation or document, by storing its notes more compactly or discarding the least useful ones.
Techniques that reduce the memory footprint of the KV cacheKV cache, n.: A memory store where a chatbot keeps its working notes on everything already read and written, so each new word does not require re-reading the whole conversation from scratch. during inference, allowing longer contexts and larger batches. The main families are: architectural changes that store fewer or smaller key–value vectors per token, such as multi-query attentionmulti-query attention, n.: A cheaper form of attention in which all the attention heads share one store of information about earlier words instead of each keeping its own, making text generation faster and lighter on memory., grouped-query attentiongrouped-query attention, n.: A middle ground between full multi-head attention and its cheapest variant: attention heads are split into groups, and each group shares one store of information about earlier words, saving memory with little loss in quality., multi-head latent attentionmulti-head latent attention, n.: An attention design from the Chinese lab DeepSeek that compresses the stored information about earlier words into a much smaller summary, so long conversations need far less memory. and cross-layer sharing, which must be built in during training; quantizing cached keys and values to 8, 4 or even 2 bits; eviction policies that keep only the tokens likely to matter, for example those that have received the most attention ('heavy hitters'), a few initial attention-sinkattention sink, n.: A quirk of language models: they pour a large share of their attention onto the very first word of the text, even when it means nothing, as a place to park attention they do not need. tokens plus a recent window, or tokens chosen per head from an observation window; and merging similar entries. Post-hoc methods trade memory for some accuracy, a loss often invisible on short benchmarks but apparent on long contextlong context, n.: The ability of an AI model to work with very long inputs — whole books, codebases or hours of transcripts — in one go, and the techniques that make this possible. retrieval and reasoning.
Quantizing the KV cache to 4 bits let us double the context length on the same GPU.
Origin
Grew from multi-query attention (Noam Shazeer, 2019) and grouped-query attention (2023). Post-hoc compression became a research field in 2023–2024 with H2O's heavy-hitter eviction (Zhenyu Zhang and colleagues, 2023), StreamingLLM's attention sinks (Guangxuan Xiao and colleagues, 2023) and low-bit KV-cache quantization methods such as KIVI (2024).
Related
- KV cacheKV cache, n.: A memory store where a chatbot keeps its working notes on everything already read and written, so each new word does not require re-reading the whole conversation from scratch.
- grouped-query attentiongrouped-query attention, n.: A middle ground between full multi-head attention and its cheapest variant: attention heads are split into groups, and each group shares one store of information about earlier words, saving memory with little loss in quality.
- multi-head latent attentionmulti-head latent attention, n.: An attention design from the Chinese lab DeepSeek that compresses the stored information about earlier words into a much smaller summary, so long conversations need far less memory.
- attention sinkattention sink, n.: A quirk of language models: they pour a large share of their attention onto the very first word of the text, even when it means nothing, as a place to park attention they do not need.
- quantizationquantization, n.: Storing an AI model's numbers with fewer digits, like rounding prices to the nearest dollar, so the model takes less memory and runs faster, usually with only a small loss of accuracy.
- long contextlong context, n.: The ability of an AI model to work with very long inputs — whole books, codebases or hours of transcripts — in one go, and the techniques that make this possible.
Mentioned in
KV cacheKV cache, n.: A memory store where a chatbot keeps its working notes on everything already read and written, so each new word does not require re-reading the whole conversation from scratch.