batch inference
In plain wordsRunning an AI model over a big pile of inputs as one job rather than answering each request live, like processing a stack of forms overnight. Results come back later but cost less.
The running of a model over a large set of inputs as an offline job whose results are collected later, rather than answering requests interactively. Because nobody waits on any single result, the system can use very large batches, fill accelerators to capacity, shift work into off-peak hours or spare capacity and retry failures, maximizing throughputthroughput, n.: How much work an AI system gets done per second across all its users at once — for a chatbot service, roughly how much text it can produce in total each second. and minimizing cost per item at the expense of latencylatency, n.: How long you wait between asking an AI system for something and getting the answer. Lower latency means a snappier, more responsive system.. Classic examples are nightly scoring of every customer with a churn model, embedding a document collection for vector searchvector search, n.: Searching by closeness instead of exact match: the query and all stored items are lists of numbers, and the search returns the items whose numbers are most similar to the query's., labeling or filtering datasets, and running evaluations. Many LLM APIsmodel API, n.: A web service through which programs send requests to an AI model run by a company and receive its answers, usually paying per use instead of running the model themselves. offer batch endpoints that accept a file of requests and return results within a window such as 24 hours at about half the interactive price.
We re-embed the whole product catalog with batch inference every Sunday night at half the API price.
Origin
Rooted in the batch processing of mainframe computing; 'batch prediction' and 'batch scoring' were standard MLOps terms by the late 2010s, contrasted with 'online' or 'real-time' inference. OpenAI introduced an LLM Batch API with a 50% discount and 24-hour turnaround in April 2024, and other model providers soon offered similar endpoints.
Usage note
Not to be confused with batching inside a live server (continuous batchingcontinuous batching, n.: A way for an AI server to handle many conversations together, letting new ones join and finished ones leave after every word, instead of waiting for a whole group to finish.) or with batch sizebatch size, n.: How many training examples a model looks at before each adjustment of its settings. Bigger batches use hardware efficiently but bring diminishing returns; smaller ones are noisier but cheaper per step. in training. Some cloud platforms use 'asynchronous inference' for a middle ground: queued requests answered within minutes.
Related
- model servingmodel serving, n.: Running a trained AI model as a live service, so that apps and people can send it requests and get answers back quickly, reliably and at a sensible cost.
- throughputthroughput, n.: How much work an AI system gets done per second across all its users at once — for a chatbot service, roughly how much text it can produce in total each second.
- inference costinference cost, n.: What it costs to use an AI model after it has been built: the computing power, electricity and hardware needed every time it answers, usually billed per chunk of text.
- continuous batchingcontinuous batching, n.: A way for an AI server to handle many conversations together, letting new ones join and finished ones leave after every word, instead of waiting for a whole group to finish.
- model APImodel API, n.: A web service through which programs send requests to an AI model run by a company and receive its answers, usually paying per use instead of running the model themselves.
- MLOpsMLOps, n.: The working habits and tools that keep machine-learning models dependable inside real products once the experiment is over: automated testing, release, monitoring and retraining; the factory and maintenance crew around a model, not the model itself.
- model servingmodel serving, n.: Running a trained AI model as a live service, so that apps and people can send it requests and get answers back quickly, reliably and at a sensible cost.
- continuous batchingcontinuous batching, n.: A way for an AI server to handle many conversations together, letting new ones join and finished ones leave after every word, instead of waiting for a whole group to finish.
Mentioned in
inference costinference cost, n.: What it costs to use an AI model after it has been built: the computing power, electricity and hardware needed every time it answers, usually billed per chunk of text., model APImodel API, n.: A web service through which programs send requests to an AI model run by a company and receive its answers, usually paying per use instead of running the model themselves., continuous batchingcontinuous batching, n.: A way for an AI server to handle many conversations together, letting new ones join and finished ones leave after every word, instead of waiting for a whole group to finish., mini-batchmini-batch, n.: A small random handful of training examples that a model learns from in a single step. Training works through many such handfuls instead of studying the whole dataset at once., model deploymentmodel deployment, n.: Putting a trained AI model to work in a real product so that people or other software can use it — and doing so carefully enough that the change can be undone if something goes wrong., model servingmodel serving, n.: Running a trained AI model as a live service, so that apps and people can send it requests and get answers back quickly, reliably and at a sensible cost., nondeterminismnondeterminism, n.: The fact that an AI model can give different answers to exactly the same question, partly because it deliberately chooses words with some randomness and partly because of tiny computing differences., rate limitrate limit, n.: A cap on how much a customer may use an online AI service in a given time — so many requests or words per minute — beyond which extra requests are refused until later., streamingstreaming, n.: Sending an AI's answer piece by piece while it is still being written, so the words appear on screen one after another instead of all at once at the end., offloadingoffloading, n.: Moving part of an AI model's data or work out of a chip's scarce fast memory onto ordinary memory, a disk or another computer, so that models too big for the chip can still run., RayRay, prop. n.: Free software that spreads Python programs, including AI training and model serving, across many computers and graphics chips so that big jobs run in parallel.