Hardware & ComputeInference & EfficiencyTraining & Post-training

memory wall

nounAlso: memory gap, processor–memory gap

In plain wordsThe growing gap between how fast chips can calculate and how fast they can fetch data from memory. Processors increasingly sit idle, waiting for numbers to arrive.

The performance limit that arises because processor arithmetic throughput has grown much faster than the bandwidth and latency of off-chip memory, so that more and more workloads are bound by data movement rather than computation. In AI the gap is stark: one analysis found that peak server FLOPS grew about 3.0× every two years over two decades, against 1.6× for DRAM bandwidth and 1.4× for interconnect bandwidth. Responses span the stack: high-bandwidth memoryhigh-bandwidth memory, n.: Very fast memory made by stacking memory chips on top of each other and placing them right next to the processor, so AI chips can be fed data quickly enough. and advanced packagingadvanced packaging, n.: Ways of joining several chips very closely inside one package — side by side on a silicon base or stacked on top of each other — so they act almost like one bigger chip.; large on-chip SRAMSRAM, abbr.: The fastest kind of computer memory, built directly into processor chips. It is quick but takes up a lot of space, so chips hold only a little of it. and wafer-scalewafer-scale engine, n.: A single giant processor made from nearly a whole dinner-plate-sized silicon wafer instead of the small chips normally cut from it, so data rarely has to leave the chip. designs; analog in-memory computinganalog in-memory computing, n.: Doing AI math inside the memory itself, using electrical currents through stored values, instead of shuttling numbers back and forth to a separate processor. It could save lots of energy but is imprecise.; and algorithms that raise arithmetic intensityarithmetic intensity, n.: How much calculating a task does for each piece of data it fetches from memory. Tasks with little calculation per fetch are slowed by memory, not by the chip's math speed. or shrink data, such as FlashAttentionFlashAttention, prop. n.: A faster way for AI chips to compute attention, the step in which a language model relates every word to every other. It gets exactly the same answer while moving far less data around the chip., batching, quantizationquantization, n.: Storing an AI model's numbers with fewer digits, like rounding prices to the nearest dollar, so the model takes less memory and runs faster, usually with only a small loss of accuracy. and mixture of expertsmixture of experts, n.: A way to build very large AI models cheaply: the model contains many specialist sub-networks, and for each word only a few are switched on, picked by a small dispatcher network.. Moving data also costs far more energy than computing on it, so the memory wall is a power problem as well as a speed problem.

Origin

Coined by William A. Wulf and Sally A. McKee in 'Hitting the Memory Wall: Implications of the Obvious' (ACM SIGARCH Computer Architecture News, March 1995), which extrapolated that processor speed growth would make memory access dominate run time. Amir Gholami and colleagues revived the phrase for AI in 'AI and Memory Wall' (2021; arXiv 2024).

Usage note

Related to, but distinct from, the von Neumann bottleneckvon Neumann bottleneck, n.: The traffic jam caused by ordinary computers keeping data and processing in separate places, so information must constantly shuttle back and forth through one narrow channel., which names the single channel between processor and memory in stored-program computers rather than a growth-rate gap.

Sources

Wulf, W. A. & McKee, S. A. (1995). Hitting the Memory Wall: Implications of the Obvious. ACM SIGARCH Computer Architecture News 23(1).
Gholami, A., Yao, Z., Kim, S., Hooper, C., Mahoney, M. W. & Keutzer, K. (2024). AI and Memory Wall. arXiv:2403.14123. arxiv.org/abs/2403.14123

Mentioned in

high-bandwidth memoryhigh-bandwidth memory, n.: Very fast memory made by stacking memory chips on top of each other and placing them right next to the processor, so AI chips can be fed data quickly enough., interconnectinterconnect, n.: The wiring and networking that connect chips to each other — inside a server, across a data center, or within a single chip — so they can share data and work as one., memory bandwidthmemory bandwidth, n.: How fast data can be moved between a chip's memory and its calculators. For chatbots, this often decides how quickly words appear, more than raw calculating speed., analog in-memory computinganalog in-memory computing, n.: Doing AI math inside the memory itself, using electrical currents through stored values, instead of shuttling numbers back and forth to a separate processor. It could save lots of energy but is imprecise., arithmetic intensityarithmetic intensity, n.: How much calculating a task does for each piece of data it fetches from memory. Tasks with little calculation per fetch are slowed by memory, not by the chip's math speed., CerebrasCerebras, prop. n.: An American chip company that makes a single giant processor the size of a whole silicon wafer, used to train and run AI models very quickly., DRAMDRAM, abbr.: The main working memory of computers and AI chips: dense and fairly cheap, but slower than on-chip memory and needing constant refreshing to keep its data., memory-boundmemory-bound, adj.: Describes a computer task slowed by how fast data can be fetched from memory rather than by how fast the chip can calculate — like a fast cook kept waiting for deliveries., offloadingoffloading, n.: Moving part of an AI model's data or work out of a chip's scarce fast memory onto ordinary memory, a disk or another computer, so that models too big for the chip can still run., roofline modelroofline model, n.: A simple chart showing the fastest a program can run on a chip: limited either by how quickly data arrives from memory or by the chip's top calculating speed., SRAMSRAM, abbr.: The fastest kind of computer memory, built directly into processor chips. It is quick but takes up a lot of space, so chips hold only a little of it., von Neumann bottleneckvon Neumann bottleneck, n.: The traffic jam caused by ordinary computers keeping data and processing in separate places, so information must constantly shuttle back and forth through one narrow channel., wafer-scale enginewafer-scale engine, n.: A single giant processor made from nearly a whole dinner-plate-sized silicon wafer instead of the small chips normally cut from it, so data rarely has to leave the chip.