PagedAttention

A memory-management scheme from the vLLMvLLM, prop. n.: A free, open-source program for running large language models on servers efficiently, so one machine can answer many more users at once. Many companies use it to host AI models. project that stores each sequence's KV cacheKV cache, n.: A memory store where a chatbot keeps its working notes on everything already read and written, so each new word does not require re-reading the whole conversation from scratch. in fixed-size blocks (for example 16 tokens' worth of keys and values) scattered anywhere in GPU memory, with a per-sequence block table translating token positions to physical blocks, much as an operating system's page table maps virtual memory onto physical pages; the attentionattention, n.: A way for an AI model to choose what to focus on. For each word it processes, the model scores how relevant every other word is and draws information mostly from the relevant ones. kernel reads keys and values through this table. Blocks are allocated on demand as a sequence grows, avoiding the fragmentation and over-reservation of systems that pre-allocate space for the maximum possible length: its authors measured 60–80% of KV-cache memory wasted in earlier systems versus under 4% with paging. Blocks can be shared copy-on-write among sequences with a common prefix, helping parallel sampling, beam searchbeam search, n.: A search method that keeps several of the best partial answers at each step, rather than only the single best, to find a better overall result without trying every possibility. and prompt cachingprompt caching, n.: Saving the work a model did reading the start of a prompt, so later requests that begin the same way can skip that reading and come back faster and cheaper.. In the authors' tests, the larger batches that then fit raised serving throughputthroughput, n.: How much work an AI system gets done per second across all its users at once — for a chatbot service, roughly how much text it can produce in total each second. two- to fourfold at equal latency. Paged KV caches are now standard in LLM serving enginesmodel serving, n.: Running a trained AI model as a live service, so that apps and people can send it requests and get answers back quickly, reliably and at a sensible cost..

With PagedAttention the same GPU held three times as many concurrent conversations.