memory-bound
In plain wordsDescribes a computer task slowed by how fast data can be fetched from memory rather than by how fast the chip can calculate — like a fast cook kept waiting for deliveries.
Describing a workload whose speed is limited by memory bandwidthmemory bandwidth, n.: How fast data can be moved between a chip's memory and its calculators. For chatbots, this often decides how quickly words appear, more than raw calculating speed., the rate at which data moves between memory (such as a GPU's high-bandwidth memoryhigh-bandwidth memory, n.: Very fast memory made by stacking memory chips on top of each other and placing them right next to the processor, so AI chips can be fed data quickly enough.) and the processor, rather than by peak arithmetic rate. In the roofline modelroofline model, n.: A simple chart showing the fastest a program can run on a chip: limited either by how quickly data arrives from memory or by the chip's top calculating speed., a kernel is memory-bound when its arithmetic intensityarithmetic intensity, n.: How much calculating a task does for each piece of data it fetches from memory. Tasks with little calculation per fetch are slowed by memory, not by the chip's math speed. (operations per byte moved) falls below the hardware's ratio of peak FLOPSFLOPS, abbr.: Floating-point operations per second: how many arithmetic steps a computer can do each second. It is the standard speed rating for supercomputers and AI chips. to bandwidth, a few hundred operations per byte on current AI accelerators; above that ratio it is compute-bound. In large language modellarge language model, n.: A computer program trained on enormous amounts of text to predict the words that come next. At vast scale, that simple skill lets it chat, answer questions, translate, summarize and write code. inference the decode phasedecode phase, n.: The second stage of an AI's reply, in which it writes the answer one word-piece at a time, each step depending on everything written so far. at small batch sizes is the classic memory-bound case, because every weight is read to produce a single token per sequence, whereas prefillprefill, n.: The first stage of an AI's reply, in which it reads your whole prompt in one go before it starts writing. Long prompts make this stage take longer. and training are usually compute-bound. Remedies include batching, quantizationquantization, n.: Storing an AI model's numbers with fewer digits, like rounding prices to the nearest dollar, so the model takes less memory and runs faster, usually with only a small loss of accuracy., kernel fusionkernel fusion, n.: Merging several small calculation steps into one, so a chip finishes them in a single pass instead of saving and reloading data in between, like cooking at the counter without returning ingredients to the fridge after each step. and IO-aware algorithms such as FlashAttentionFlashAttention, prop. n.: A faster way for AI chips to compute attention, the step in which a language model relates every word to every other. It gets exactly the same answer while moving far less data around the chip. that cut memory traffic.
Decode at batch size one is memory-bound, so the 4-bit model runs nearly three times faster.
Origin
A long-standing term in high-performance computing, given a quantitative frame by the roofline model of Samuel Williams, Andrew Waterman and David Patterson (2009). It entered everyday LLM-engineering vocabulary around 2022–2023, when practitioners explained why token generation leaves GPUs underused.
Usage note
Memory-bound concerns bandwidth, not capacity: a model can fit easily in memory and still be memory-bound. Running out of memory capacity is a separate constraint, addressed by offloadingoffloading, n.: Moving part of an AI model's data or work out of a chip's scarce fast memory onto ordinary memory, a disk or another computer, so that models too big for the chip can still run., sharding or compression.
Related
- arithmetic intensityarithmetic intensity, n.: How much calculating a task does for each piece of data it fetches from memory. Tasks with little calculation per fetch are slowed by memory, not by the chip's math speed.
- roofline modelroofline model, n.: A simple chart showing the fastest a program can run on a chip: limited either by how quickly data arrives from memory or by the chip's top calculating speed.
- memory bandwidthmemory bandwidth, n.: How fast data can be moved between a chip's memory and its calculators. For chatbots, this often decides how quickly words appear, more than raw calculating speed.
- memory wallmemory wall, n.: The growing gap between how fast chips can calculate and how fast they can fetch data from memory. Processors increasingly sit idle, waiting for numbers to arrive.
- decode phasedecode phase, n.: The second stage of an AI's reply, in which it writes the answer one word-piece at a time, each step depending on everything written so far.
- prefillprefill, n.: The first stage of an AI's reply, in which it reads your whole prompt in one go before it starts writing. Long prompts make this stage take longer.
Sources
- Williams, S., Waterman, A. & Patterson, D. (2009). Roofline: An Insightful Visual Performance Model for Multicore Architectures. Communications of the ACM 52(4).
- Agarwal, M. et al. (2023). LLM Inference Performance Engineering: Best Practices. Databricks blog. databricks.com/blog/llm-inference-performance-engineering-best-practices
Mentioned in
decode phasedecode phase, n.: The second stage of an AI's reply, in which it writes the answer one word-piece at a time, each step depending on everything written so far., FlashAttentionFlashAttention, prop. n.: A faster way for AI chips to compute attention, the step in which a language model relates every word to every other. It gets exactly the same answer while moving far less data around the chip., prefillprefill, n.: The first stage of an AI's reply, in which it reads your whole prompt in one go before it starts writing. Long prompts make this stage take longer., quantizationquantization, n.: Storing an AI model's numbers with fewer digits, like rounding prices to the nearest dollar, so the model takes less memory and runs faster, usually with only a small loss of accuracy., speculative decodingspeculative decoding, n.: A speed-up trick in which a small, fast model guesses the next few words and the big model checks all the guesses at once, keeping the right ones: faster answers, same quality., chunked prefillchunked prefill, n.: A serving trick that splits the reading of a long prompt into smaller pieces and slots them between other users' replies, so nobody's answer freezes while a huge document is read., disaggregated servingdisaggregated serving, n.: Running the two stages of an AI's reply, reading the prompt and writing the answer, on separate groups of chips, each tuned for its own job, and handing the work between them., kernel fusionkernel fusion, n.: Merging several small calculation steps into one, so a chip finishes them in a single pass instead of saving and reloading data in between, like cooking at the counter without returning ingredients to the fridge after each step.