continuous batching
In plain wordsA way for an AI server to handle many conversations together, letting new ones join and finished ones leave after every word, instead of waiting for a whole group to finish.
A scheduling technique for autoregressiveautoregressive model, n.: A model that produces a sequence one piece at a time, each new piece depending on the pieces already produced — like writing a sentence word by word without going back. model servingmodel serving, n.: Running a trained AI model as a live service, so that apps and people can send it requests and get answers back quickly, reliably and at a sensible cost. that decides afresh at every generation step (iteration) which sequences make up the batch, rather than fixing the batch once per group of requests. The older static approach holds a batch together until its longest output is complete, so finished requests sit in their slots and new arrivals queue; with continuous batching a sequence exits as soon as it emits its stop token and a waiting request is admitted at the following step, so the accelerator never idles for lack of work. Because sequences in a batch sit at different positions and lengths, attentionattention, n.: A way for an AI model to choose what to focus on. For each word it processes, the model scores how relevant every other word is and draws information mostly from the relevant ones. is computed per sequence while the large matrix multiplications are batched across all tokens. Combined with PagedAttentionPagedAttention, prop. n.: A way of storing an AI model's working memory in small fixed-size blocks, like the pages of a computer's memory, so little space is wasted and more users can be served at once.-style memory management and chunked prefillchunked prefill, n.: A serving trick that splits the reading of a long prompt into smaller pieces and slots them between other users' replies, so nobody's answer freezes while a huge document is read., it multiplied LLM serving throughputthroughput, n.: How much work an AI system gets done per second across all its users at once — for a chatbot service, roughly how much text it can produce in total each second. (an Anyscale benchmark in 2023 reported up to 23 times the throughput of naive static batching) while lowering median latencylatency, n.: How long you wait between asking an AI system for something and getting the answer. Lower latency means a snappier, more responsive system., and it is the default in production engines.
After we enabled continuous batching, short replies stopped waiting behind 4,000-token essays.
Origin
Introduced as 'iteration-level scheduling', together with 'selective batching', by Gyeong-In Yu and colleagues at Seoul National University and FriendliAI in the Orca serving system (OSDI 2022), which reported a 36.9-fold throughput gain over NVIDIA FasterTransformer on GPT-3 175B. The name 'continuous batching' spread in 2023 through Hugging Face's Text Generation Inference, vLLM and an Anyscale blog post.
Usage note
'Dynamic batching' in older serving systems usually means grouping requests that arrive within a short time window, at the level of whole requests; it is not the same technique, although some writers use the terms interchangeably. NVIDIA calls its version in-flight batching.
Related
- model servingmodel serving, n.: Running a trained AI model as a live service, so that apps and people can send it requests and get answers back quickly, reliably and at a sensible cost.
- PagedAttentionPagedAttention, prop. n.: A way of storing an AI model's working memory in small fixed-size blocks, like the pages of a computer's memory, so little space is wasted and more users can be served at once.
- vLLMvLLM, prop. n.: A free, open-source program for running large language models on servers efficiently, so one machine can answer many more users at once. Many companies use it to host AI models.
- chunked prefillchunked prefill, n.: A serving trick that splits the reading of a long prompt into smaller pieces and slots them between other users' replies, so nobody's answer freezes while a huge document is read.
- throughputthroughput, n.: How much work an AI system gets done per second across all its users at once — for a chatbot service, roughly how much text it can produce in total each second.
- batch sizebatch size, n.: How many training examples a model looks at before each adjustment of its settings. Bigger batches use hardware efficiently but bring diminishing returns; smaller ones are noisier but cheaper per step.
- batch inferencebatch inference, n.: Running an AI model over a big pile of inputs as one job rather than answering each request live, like processing a stack of forms overnight. Results come back later but cost less.
Sources
- Yu, G.-I., Jeong, J. S., Kim, G.-W., Kim, S. & Chun, B.-G. (2022). Orca: A Distributed Serving System for Transformer-Based Generative Models. OSDI 2022. usenix.org/conference/osdi22/presentation/yu
- Daniel, C., Shen, C., Liang, E. & Liaw, R. (2023). How continuous batching enables 23x throughput in LLM inference while reducing p50 latency. Anyscale blog, 22 June 2023. anyscale.com/blog/continuous-batching-llm-inference
Mentioned in
batch inferencebatch inference, n.: Running an AI model over a big pile of inputs as one job rather than answering each request live, like processing a stack of forms overnight. Results come back later but cost less., decode phasedecode phase, n.: The second stage of an AI's reply, in which it writes the answer one word-piece at a time, each step depending on everything written so far., inference engineinference engine, n.: The part of an AI system that does the working-out: in older AI, the program that applies rules to facts; in today's AI, the software that runs a trained model to produce answers., llama.cppllama.cpp, prop. n.: A free, open-source program that runs AI language models on ordinary computers and phones, without needing a powerful graphics card, by using compact low-precision versions of the models., mini-batchmini-batch, n.: A small random handful of training examples that a model learns from in a single step. Training works through many such handfuls instead of studying the whole dataset at once., model servingmodel serving, n.: Running a trained AI model as a live service, so that apps and people can send it requests and get answers back quickly, reliably and at a sensible cost., nondeterminismnondeterminism, n.: The fact that an AI model can give different answers to exactly the same question, partly because it deliberately chooses words with some randomness and partly because of tiny computing differences., throughputthroughput, n.: How much work an AI system gets done per second across all its users at once — for a chatbot service, roughly how much text it can produce in total each second., vLLMvLLM, prop. n.: A free, open-source program for running large language models on servers efficiently, so one machine can answer many more users at once. Many companies use it to host AI models., chunked prefillchunked prefill, n.: A serving trick that splits the reading of a long prompt into smaller pieces and slots them between other users' replies, so nobody's answer freezes while a huge document is read., PagedAttentionPagedAttention, prop. n.: A way of storing an AI model's working memory in small fixed-size blocks, like the pages of a computer's memory, so little space is wasted and more users can be served at once., SGLangSGLang, prop. n.: A free, open-source engine for running AI language and image models on servers, built to be fast when many requests begin the same way, as in chat and agent workloads., TensorRT-LLMTensorRT-LLM, prop. n.: NVIDIA's free software library for running large language models as fast as possible on NVIDIA's own graphics chips, used by many companies that host AI models.