chunked prefill
In plain wordsA serving trick that splits the reading of a long prompt into smaller pieces and slots them between other users' replies, so nobody's answer freezes while a huge document is read.
A scheduling technique that spreads the prefillprefill, n.: The first stage of an AI's reply, in which it reads your whole prompt in one go before it starts writing. Long prompts make this stage take longer. of a long prompt over several iterations, one slice of tokens at a time, and fills the rest of each iteration's batch with decodedecode phase, n.: The second stage of an AI's reply, in which it writes the answer one word-piece at a time, each step depending on everything written so far. steps of other running requests. A batch of prefill alone saturates arithmetic while a batch of decodes alone is memory-boundmemory-bound, adj.: Describes a computer task slowed by how fast data can be fetched from memory rather than by how fast the chip can calculate — like a fast cook kept waiting for deliveries.; combining a prefill slice with many decodes keeps both resources busy, so the decodes ride along at little extra cost, the effect the SARATHI authors called piggybacking. Above all it caps how long any single iteration can take, so a 100,000-token prompt no longer stalls every other user's token stream, cutting tail time per output tokentime per output token, n.: Once an AI has started replying, the average wait between one piece of its answer and the next. It sets how quickly the text seems to flow. at a modest cost in time to first tokentime to first token, n.: How long a chatbot takes to start answering: the pause between pressing enter and seeing the first word appear. for the chunked request; chunk size sets the trade-off. It is a standard option in vLLMvLLM, prop. n.: A free, open-source program for running large language models on servers efficiently, so one machine can answer many more users at once. Many companies use it to host AI models., SGLangSGLang, prop. n.: A free, open-source engine for running AI language and image models on servers, built to be fast when many requests begin the same way, as in chat and agent workloads. and TensorRT-LLM, and the main alternative to separating the two phases through disaggregated servingdisaggregated serving, n.: Running the two stages of an AI's reply, reading the prompt and writing the answer, on separate groups of chips, each tuned for its own job, and handing the work between them..
Turning on chunked prefill with 2,048-token chunks removed the stutters users saw during long uploads.
Origin
Introduced by Amey Agrawal (Georgia Tech) and colleagues at Microsoft Research India in SARATHI (arXiv, August 2023), which paired 'chunked prefills' with 'decode-maximal batching'; the follow-up Sarathi-Serve appeared at OSDI 2024. DeepSpeed-FastGen announced a similar scheme, 'Dynamic SplitFuse', in late 2023.
Related
- prefillprefill, n.: The first stage of an AI's reply, in which it reads your whole prompt in one go before it starts writing. Long prompts make this stage take longer.
- continuous batchingcontinuous batching, n.: A way for an AI server to handle many conversations together, letting new ones join and finished ones leave after every word, instead of waiting for a whole group to finish.
- disaggregated servingdisaggregated serving, n.: Running the two stages of an AI's reply, reading the prompt and writing the answer, on separate groups of chips, each tuned for its own job, and handing the work between them.
- time per output tokentime per output token, n.: Once an AI has started replying, the average wait between one piece of its answer and the next. It sets how quickly the text seems to flow.
- vLLMvLLM, prop. n.: A free, open-source program for running large language models on servers efficiently, so one machine can answer many more users at once. Many companies use it to host AI models.
Sources
- Agrawal, A., Panwar, A., Mohan, J., Kwatra, N., Gulavani, B. S. & Ramjee, R. (2023). SARATHI: Efficient LLM Inference by Piggybacking Decodes with Chunked Prefills. arXiv:2308.16369. arxiv.org/abs/2308.16369
Mentioned in
continuous batchingcontinuous batching, n.: A way for an AI server to handle many conversations together, letting new ones join and finished ones leave after every word, instead of waiting for a whole group to finish., prefillprefill, n.: The first stage of an AI's reply, in which it reads your whole prompt in one go before it starts writing. Long prompts make this stage take longer., vLLMvLLM, prop. n.: A free, open-source program for running large language models on servers efficiently, so one machine can answer many more users at once. Many companies use it to host AI models., disaggregated servingdisaggregated serving, n.: Running the two stages of an AI's reply, reading the prompt and writing the answer, on separate groups of chips, each tuned for its own job, and handing the work between them., SGLangSGLang, prop. n.: A free, open-source engine for running AI language and image models on servers, built to be fast when many requests begin the same way, as in chat and agent workloads., time per output tokentime per output token, n.: Once an AI has started replying, the average wait between one piece of its answer and the next. It sets how quickly the text seems to flow.