Inference & Efficiency

Running models fast and cheaply: serving, caching, quantization. 52 entries live here, and 112 from other domains touch it.

Essential

inference costWhat it costs to use an AI model after it has been built: the computing power, electricity and hardware needed every time it answers, usually billed per chunk of text.
on-device AIAI that runs directly on your own phone, laptop, car or headset rather than on a company's distant servers, so it can work offline, respond instantly and keep your data on the device.

All other entries

active parametersThe part of a model's internal numbers actually used to process each word. Some big models use only a small slice of themselves at a time, making them cheaper to run than their size suggests.
AWQA method for shrinking large language models to about 4 bits per number that protects the small share of numbers most important to the model, judged by the data flowing through them.
batch inferenceRunning an AI model over a big pile of inputs as one job rather than answering each request live, like processing a stack of forms overnight. Results come back later but cost less.
BitNetA family of AI language models from Microsoft in which most of the learned numbers are limited to minus one, zero or one, so they need far less memory and energy to run.
chunked prefillA serving trick that splits the reading of a long prompt into smaller pieces and slots them between other users' replies, so nobody's answer freezes while a huge document is read.
continuous batchingA way for an AI server to handle many conversations together, letting new ones join and finished ones leave after every word, instead of waiting for a whole group to finish.
decode phaseThe second stage of an AI's reply, in which it writes the answer one word-piece at a time, each step depending on everything written so far.
disaggregated servingRunning the two stages of an AI's reply, reading the prompt and writing the answer, on separate groups of chips, each tuned for its own job, and handing the work between them.
early exitLetting an AI model stop partway through its layers and answer early when it is already confident, saving computing effort on easy inputs.
edge AIDoing AI's computing on or near the devices that collect the data, such as cameras, cars, factory machines, phones or local servers, instead of sending everything off to distant data centers.
FlashAttentionA faster way for AI chips to compute attention, the step in which a language model relates every word to every other. It gets exactly the same answer while moving far less data around the chip.
GGUFA file format that packs an AI language model, often in a shrunken low-precision version, into one file that ordinary computers can load and run quickly; used by llama.cpp and Ollama.
GPTQA popular method for shrinking large language models to about 4 bits per number after training, carefully adjusting the remaining numbers to make up for each rounding error.
inference providerA company that runs AI models on its own computers and sells access to them over the internet, charging by use, so customers need not buy or operate the hardware themselves.
kernel fusionMerging several small calculation steps into one, so a chip finishes them in a single pass instead of saving and reloading data in between, like cooking at the counter without returning ingredients to the fridge after each step.
KV cacheA memory store where a chatbot keeps its working notes on everything already read and written, so each new word does not require re-reading the whole conversation from scratch.
KV cache compressionTechniques that shrink the memory an AI model uses to remember a long conversation or document, by storing its notes more compactly or discarding the least useful ones.
latencyHow long you wait between asking an AI system for something and getting the answer. Lower latency means a snappier, more responsive system.
llama.cppA free, open-source program that runs AI language models on ordinary computers and phones, without needing a powerful graphics card, by using compact low-precision versions of the models.
LLM.int8()An early, widely used method for running very large language models with 8-bit numbers, roughly halving their memory needs, by keeping a few unusually large values at higher precision.
memory-boundDescribes a computer task slowed by how fast data can be fetched from memory rather than by how fast the chip can calculate — like a fast cook kept waiting for deliveries.
model cascadeA chain of AI models ordered from cheapest to most capable: each request goes to the cheap one first and moves up the chain only if the answer does not look good enough.
model compressionThe family of techniques for making AI models smaller, faster or cheaper to run while keeping almost all of their accuracy.
model routingAutomatically choosing which AI model should answer each request, a cheap quick one for easy questions and a bigger slower one for hard ones, to save money without hurting quality.
model servingRunning a trained AI model as a live service, so that apps and people can send it requests and get answers back quickly, reliably and at a sensible cost.
N:M sparsityA tidy pattern of zeros inside an AI model, such as two zeros in every group of four numbers, that special chip hardware can skip to run the model faster.
offloadingMoving part of an AI model's data or work out of a chip's scarce fast memory onto ordinary memory, a disk or another computer, so that models too big for the chip can still run.
OllamaA free app that makes it easy to download and run AI language models on your own computer with a single command, keeping your data on your machine.
PagedAttentionA way of storing an AI model's working memory in small fixed-size blocks, like the pages of a computer's memory, so little space is wasted and more users can be served at once.
post-training quantizationShrinking an already-trained AI model by storing its numbers in fewer bits, without training it again; a small sample of example inputs is usually enough to fine-tune the rounding.
prefillThe first stage of an AI's reply, in which it reads your whole prompt in one go before it starts writing. Long prompts make this stage take longer.
prompt cachingSaving the work a model did reading the start of a prompt, so later requests that begin the same way can skip that reading and come back faster and cheaper.
pruningCutting away the parts of an AI model that contribute least, as a gardener trims branches, so the model becomes smaller or faster while still working nearly as well.
quantizationStoring an AI model's numbers with fewer digits, like rounding prices to the nearest dollar, so the model takes less memory and runs faster, usually with only a small loss of accuracy.
quantization-aware trainingTraining or fine-tuning an AI model while simulating the low-precision numbers it will use later, so that it learns to work well despite the rounding.
SGLangA free, open-source engine for running AI language and image models on servers, built to be fast when many requests begin the same way, as in chat and agent workloads.
sleep-time computeLetting an AI think about your documents or project while it would otherwise sit idle, before you ask anything, so it can answer later questions faster and more cheaply.
SmoothQuantA method that lets large language models run entirely on 8-bit numbers by moving hard-to-round extreme values out of the data flowing through the model and into its fixed numbers, which cope better.
sparsityThe property of being mostly zeros or mostly switched off. AI systems exploit it by skipping the empty parts, so they need less memory and computation.
speculative decodingA speed-up trick in which a small, fast model guesses the next few words and the big model checks all the guesses at once, keeping the right ones: faster answers, same quality.
streamingSending an AI's answer piece by piece while it is still being written, so the words appear on screen one after another instead of all at once at the end.
swarm scalingGetting better AI results by putting many AI agents on a task at once instead of letting one think longer — faster, but more expensive for each gain.
TensorRTNVIDIA's software for making a trained neural network run as fast as possible on NVIDIA graphics chips, by merging and simplifying its calculations and testing which way of doing each one is quickest.
TensorRT-LLMNVIDIA's free software library for running large language models as fast as possible on NVIDIA's own graphics chips, used by many companies that host AI models.
throughputHow much work an AI system gets done per second across all its users at once — for a chatbot service, roughly how much text it can produce in total each second.
time per output tokenOnce an AI has started replying, the average wait between one piece of its answer and the next. It sets how quickly the text seems to flow.
time to first tokenHow long a chatbot takes to start answering: the pause between pressing enter and seeing the first word appear.
TinyMLMachine learning shrunk to run on tiny, cheap, battery-powered chips inside sensors, wearables or appliances, using so little power that a coin-sized battery can last for months or years.
tokens per secondA speed rating for AI text generation: how many word-pieces a model produces each second. Five to ten keeps up with a fast reader; hundreds feel instant.
vLLMA free, open-source program for running large language models on servers efficiently, so one machine can answer many more users at once. Many companies use it to host AI models.

Also relevant

112

AI acceleratorAny chip built or tuned to speed up AI — graphics chips, Google's TPUs, phone 'neural engines' and many startup designs. Such chips do the heavy math of training and running models.
AI compilerSoftware that translates an AI model into efficient instructions for a specific chip, rearranging and combining its calculations so the hardware is used as fully as possible.
AI gatewayA checkpoint between an organization's apps and the AI models they use that handles access, costs, logging, safety checks and switching between providers in one place.
analog in-memory computingDoing AI math inside the memory itself, using electrical currents through stored values, instead of shuttling numbers back and forth to a separate processor. It could save lots of energy but is imprecise.
arithmetic intensityHow much calculating a task does for each piece of data it fetches from memory. Tasks with little calculation per fetch are slowed by memory, not by the chip's math speed.
ASICA chip whose circuits are laid out to do one kind of work extremely well instead of running any program. AI companies design their own to run their models more cheaply than with off-the-shelf chips.
attention sinkA quirk of language models: they pour a large share of their attention onto the very first word of the text, even when it means nothing, as a place to park attention they do not need.
beam searchA search method that keeps several of the best partial answers at each step, rather than only the single best, to find a better overall result without trying every possibility.
best-of-n samplingAsking an AI the same question several times and keeping whichever answer a scoring model or checker rates highest — a simple way to trade extra computing for better answers.
big O notationShorthand for how the cost of a computation grows as the problem gets bigger. Saying attention costs 'O(n²)' means doubling the length of the text roughly quadruples the work.
cache-augmented generationAn alternative to search-based AI answering: load the whole knowledge base, if it is small enough, into the model's context once, save the model's processed state, and reuse it for every question.
chat templateThe exact text layout a chat AI was trained on for marking who is speaking — system, user or assistant. Software must reproduce it precisely, or the model performs noticeably worse.
classifier-free guidanceA setting that makes an image generator follow the prompt more strictly by comparing what it would draw with and without the prompt, then exaggerating the difference.
computeThe raw computing power used to train and run AI, usually measured by how many specialized chips are used and for how long. With data and algorithms, it is a main driver of AI progress.
confidential computingRunning computations inside a hardware-sealed compartment of a chip, so that data and AI models stay protected even from the cloud company or administrator operating the machine.
consistency modelA generator trained to jump from noise straight to a finished image in one step, instead of the dozens of small steps a diffusion model normally takes.
context windowHow much an AI model can read and keep in view at once: the instructions, the conversation so far and any attached documents. Text that does not fit has to be cut, summarized or left out.
Core MLApple's software framework that lets apps on iPhones, iPads and Macs run AI models directly on the device, using its own chips instead of sending data to the cloud.
CPUThe main general-purpose chip in a computer, good at running many different programs step by step. In AI systems it usually coordinates the work while specialized chips do the heavy math.
decodingThe method a language model uses to turn its predictions into actual text — for example, always picking the most likely next word, or choosing somewhat at random among the likely ones.
denoising diffusion implicit modelA faster way to run a diffusion image generator that skips most of the steps and, for the same starting noise, always gives the same picture.
depthwise separable convolutionA cheaper way to do the main operation of image networks: first scan each channel of the input separately, then mix the channels together with a tiny one-by-one step.
diffusion distillationTraining a fast image generator to imitate a slow diffusion model, so that a picture that once took fifty steps can be made in one to four.
diffusion samplerThe step-by-step routine a diffusion model follows to turn noise into a finished picture; the choice of sampler and number of steps trades speed against quality.
DRAMThe main working memory of computers and AI chips: dense and fairly cheap, but slower than on-chip memory and needing constant refreshing to keep its data.
EfficientNetA family of image-recognition networks from 2019 that grows a small base design in a balanced way, deeper, wider and sharper together, to get more accuracy for each unit of computing.
environmental impact of AIThe effects of AI on the planet: the electricity, water, materials and land used to make chips, build data centers and train and run models, and the pollution that results.
ExecuTorchMeta's free software for running PyTorch AI models directly on phones, wearables and other small devices, without needing an internet connection or a data center.
expert parallelismA way of running a 'mixture of experts' model — one made of many specialist sub-networks — by giving each chip its own set of specialists and sending each piece of text to whichever chip holds the specialists it needs.
FAISSA free software library from Meta for searching huge collections of embeddings very fast, on ordinary processors or graphics cards. Many AI search systems use it under the hood.
FLOPSFloating-point operations per second: how many arithmetic steps a computer can do each second. It is the standard speed rating for supercomputers and AI chips.
forward passThe step where a neural network takes an input and runs it through its layers, one after another, to produce an output — its answer or prediction.
FP16A 16-bit way of storing numbers that uses half the memory of the standard 32-bit format. It is fast for AI but can't represent very large or very tiny values.
FP4A tiny 4-bit number format that can represent only sixteen different values. With clever scaling tricks it lets the newest AI chips run, and even train, models faster and with less memory.
FP8An 8-bit number format that uses a quarter of the memory of the standard 32-bit format, letting new AI chips train and run models roughly twice as fast as with 16 bits.
FPGAA chip whose circuits can be rewired by software after it is made, so engineers can turn it into a custom processor — slower than a purpose-built chip, but flexible.
GemmaGoogle's family of free-to-download AI language models: smaller relatives of its Gemini systems, compact enough to run on a single computer and, in some versions, on a phone.
GPUA computer chip first designed to draw video-game graphics that turned out to be ideal for AI, because it can do thousands of simple calculations at the same time.
GPU-hourOne graphics chip working for one hour — the everyday unit for how much computing a job used and for renting AI chips by the hour.
greedy decodingThe simplest way for a language model to write: always pick the single most likely next word. It is fast and predictable but can produce dull or repetitive text.
Green AIA movement to make AI research and products use less energy and computing power, and to report those costs openly alongside accuracy.
grouped-query attentionA middle ground between full multi-head attention and its cheapest variant: attention heads are split into groups, and each group shares one store of information about earlier words, saving memory with little loss in quality.
high-bandwidth memoryVery fast memory made by stacking memory chips on top of each other and placing them right next to the processor, so AI chips can be fed data quickly enough.
homomorphic encryptionA way of locking data so that a computer can still work with it while it stays locked: the computer never sees the real numbers, and only the owner can open the answer.
hybrid architectureAn AI model design that mixes two kinds of building blocks — usually a few standard attention layers among cheaper, memory-saving layers — to keep most of the quality at lower cost on long inputs.
hybrid reasoning modelA single AI model that can either answer right away or first think at length, with the user or developer choosing how much thinking to allow.
inferenceDrawing a conclusion from evidence. In today's AI it most often means running an already-trained model to get an answer, as opposed to training it.
inference engineThe part of an AI system that does the working-out: in older AI, the program that applies rules to facts; in today's AI, the software that runs a trained model to produce answers.
INT8Storing numbers as whole numbers from −128 to 127 using 8 bits. AI models converted to this format run faster and need a quarter of the memory of 32-bit versions.
Jevons paradoxThe observation that making something more efficient can make people use more of it overall because it gets cheaper — as with coal in the 1800s, and with AI computing today.
knowledge distillationTraining a smaller or cheaper 'student' AI to copy the behavior of a larger 'teacher' AI, so the student keeps much of the teacher's skill while being faster to run.
latent diffusion modelA diffusion image generator that works on a shrunken, compressed version of each picture instead of the full-size pixels, which makes it far cheaper to train and run.
LiteRTGoogle's free software for running AI models on phones, small gadgets and microcontrollers; it was called TensorFlow Lite until 2024 and is used inside many Android and iPhone apps.
long contextThe ability of an AI model to work with very long inputs — whole books, codebases or hours of transcripts — in one go, and the techniques that make this possible.
lottery ticket hypothesisThe idea that a large, randomly started neural network contains a much smaller network inside it that, trained alone from the same starting values, would learn just as well — a winning lottery ticket.
low-rank approximationReplacing a big grid of numbers with a slimmer one that keeps most of the same information, the way a few pencil strokes can capture a face. It saves memory and computation.
memory bandwidthHow fast data can be moved between a chip's memory and its calculators. For chatbots, this often decides how quickly words appear, more than raw calculating speed.
memory wallThe growing gap between how fast chips can calculate and how fast they can fetch data from memory. Processors increasingly sit idle, waiting for numbers to arrive.
microscaling formatsNumber formats that store small groups of numbers with one shared scale factor, so each number can use very few bits while the group as a whole stays accurate.
min-p samplingA sampling method that lets the model consider any next word at least a set fraction as likely as its top choice, so the shortlist shrinks when the model is sure and grows when it is not.
mixture of expertsA way to build very large AI models cheaply: the model contains many specialist sub-networks, and for each word only a few are switched on, picked by a small dispatcher network.
MLXA free Apple software toolkit for running and training AI models efficiently on Mac computers with Apple's own chips, popular for running language models locally on a laptop.
MobileNetA family of compact image-recognition networks from Google, designed to run quickly on phones and other small devices with little loss of accuracy.
model APIA web service through which programs send requests to an AI model run by a company and receive its answers, usually paying per use instead of running the model themselves.
model deploymentPutting a trained AI model to work in a real product so that people or other software can use it — and doing so carefully enough that the change can be undone if something goes wrong.
multi-head latent attentionAn attention design from the Chinese lab DeepSeek that compresses the stored information about earlier words into a much smaller summary, so long conversations need far less memory.
multi-query attentionA cheaper form of attention in which all the attention heads share one store of information about earlier words instead of each keeping its own, making text generation faster and lighter on memory.
multi-token predictionTraining a language model to guess several upcoming words at once rather than only the next one, which can help it learn better and later write faster.
neural audio codecA neural network that squeezes sound into a very compact stream of numbers and rebuilds it, used both to send audio cheaply and to let AI models 'write' speech and music piece by piece.
nondeterminismThe fact that an AI model can give different answers to exactly the same question, partly because it deliberately chooses words with some randomness and partly because of tiny computing differences.
non-maximum suppressionA clean-up step in object detection: when the system draws several overlapping boxes around the same object, it keeps the most confident one and throws away the near-duplicates.
NPUA small AI chip, or part of a chip, built into phones, laptops and other devices so they can run AI features quickly without draining the battery or using the cloud.
NVLinkNvidia's private high-speed connection between its AI chips, much faster than ordinary computer connections, letting dozens of chips share memory and act almost like one giant chip.
ONNXA shared file format for trained AI models, so that a model built with one toolkit can be moved to and run by another, much as a PDF opens on any computer.
OpenAI-compatible APIAn AI service that accepts the same request format as OpenAI's, so software written for OpenAI's models can use other companies' or self-hosted models by changing an address and key.
overthinkingWhen an AI reasoning model spends far more thinking than a problem needs — pages of deliberation for a simple sum — wasting time and money and sometimes talking itself out of a right answer.
overtrainingDeliberately training a model on much more data than would be most efficient for its training budget, so that a smaller model reaches good quality and is cheaper to run afterwards.
parameter countHow many adjustable numbers an AI model contains — the usual measure of its size. Modern language models have billions to trillions of them.
photonic computingUsing light instead of electricity to compute or move data. Light can carry huge amounts of information with little energy, but building full computers from it is still very hard.
product quantizationA compression trick for embeddings: chop each long list of numbers into short pieces and replace each piece with the ID of its closest typical piece, shrinking storage many times over while keeping search workable.
QLoRAA memory-saving version of LoRA that squeezes the big model into a compact 4-bit form while training small add-ons, letting very large models be customized on a single graphics card.
rack-scale systemAn AI computer designed as one whole cabinet of chips, cooling and wiring, rather than separate servers, so all the chips inside can work together like one big chip.
rate limitA cap on how much a customer may use an online AI service in a given time — so many requests or words per minute — beyond which extra requests are refused until later.
recursive language modelA way of running an AI language model so it can handle enormous documents by writing small programs to search and split them, and calling copies of itself on the pieces.
repetition penaltyA setting that discourages an AI model from repeating itself by lowering the chances of words it has already used, so its text does not get stuck in loops.
roofline modelA simple chart showing the fastest a program can run on a chip: limited either by how quickly data arrives from memory or by the chip's top calculating speed.
safetensorsA file format for storing an AI model's learned numbers that cannot hide harmful programs and loads very quickly, replacing older formats that could run code when opened.
self-consistencyAsking an AI the same question many times, letting it reason differently each time, then going with the answer it reaches most often — a vote among its own attempts.
semantic cachingSaving an AI system's past answers and reusing one whenever a new question means the same as an old one, even if worded differently, to save time and money.
service-level objectiveA specific, measurable target for how well a service should perform — for example, 99.9 percent of requests answered within two seconds — used to decide when engineers must act.
Noam ShazeerAn American computer scientist who co-invented the transformer and several techniques found in nearly every large language model, co-founded Character.AI, and later returned to Google to co-lead Gemini.
sliding window attentionAttention in which each word looks only at a fixed number of nearby earlier words rather than the whole text, keeping the cost of very long inputs under control.
small AICheap, compact AI tools built for one practical job — like advising farmers or screening patients — that poorer countries can afford to run, instead of giant all-purpose models.
small language modelA language model built to be compact enough to run cheaply and quickly, even on a laptop or phone, trading some knowledge and skill for speed, privacy and lower cost.
sparse attentionAttention in which each word looks at only a selected subset of the other words instead of all of them, so very long texts can be processed at a fraction of the cost.
sponge exampleAn input crafted to make an AI system work as hard as possible — burning extra energy and time — to slow it down or run up costs, rather than to make it answer wrongly.
SRAMThe fastest kind of computer memory, built directly into processor chips. It is quick but takes up a lot of space, so chips hold only a little of it.
straight-through estimatorA shortcut that lets networks learn even when they contain steps with no useful slope, such as rounding: during learning, the computer simply pretends the step passed its input straight through.
tensor coreA small, specialized part of an AI chip that does one job — multiplying small blocks of numbers together — far faster than ordinary circuits can.
tensor parallelismSplitting the individual giant tables of numbers inside each layer of a model across several chips, so that every chip does a slice of every calculation at the same time.
test-time computeThe computing effort an AI model spends while answering a question, as opposed to while being trained. Letting a model think longer or try many answers can make it markedly smarter.
thinking budgetA limit, set by whoever uses an AI model, on how much private step-by-step reasoning it may do before answering: more thinking usually means better answers but higher cost and slower replies.
top-k samplingA way of making AI-written text varied but sensible: at each step the model picks at random, but only among its k most likely next words, such as the top 40.
top-p samplingA way of making AI-written text varied but sensible: the model picks at random among the smallest group of likely next words whose chances add up to a set share, such as 90%.
TOPSTrillions of operations per second: the speed rating usually given for AI chips in phones and laptops, often counting simple whole-number math rather than precise fractions.
TritonA free language from OpenAI for writing the fast inner routines of AI programs in Python-style code; its compiler handles hardware details that Nvidia's own tools leave to experts.
vector quantizationCompressing data by replacing each item with the nearest entry from a small catalog of representative examples, so that only the catalog number needs to be stored or sent.
voice activity detectionDetecting, moment by moment, whether someone is speaking in an audio stream, so a system knows when to listen, when to stop recording and when it is its turn to talk.
VRAMThe memory on a graphics card. For running AI at home it is the key number: the model has to fit in it, so more memory means bigger models.
wafer-scale engineA single giant processor made from nearly a whole dinner-plate-sized silicon wafer instead of the small chips normally cut from it, so data rarely has to leave the chip.
wake wordA word or phrase, such as 'Alexa' or 'Hey Google', that a smart speaker or phone listens for constantly and that switches it into full listening mode.
YOLOA family of fast object-detection models that look at an image once and immediately output boxes and labels for everything they recognize, quick enough for live video.