roofline model
In plain wordsA simple chart showing the fastest a program can run on a chip: limited either by how quickly data arrives from memory or by the chip's top calculating speed.
A visual performance model that bounds the attainable throughput of a computation on given hardware by the lesser of the peak compute rate and memory bandwidth multiplied by arithmetic intensityarithmetic intensity, n.: How much calculating a task does for each piece of data it fetches from memory. Tasks with little calculation per fetch are slowed by memory, not by the chip's math speed.. Plotted on log–log axes (performance against FLOP per byte), the bound forms a sloping roof for the memory-bound region that meets a flat roof for the compute-bound region at the ridge point, the minimum intensity needed to reach peak. Plotting measured kernels against the roofline shows how far each is from its limit and which resource to optimize; extensions add rooflines for caches, interconnectsinterconnect, n.: The wiring and networking that connect chips to each other — inside a server, across a data center, or within a single chip — so they can share data and work as one., lower-precision tensor unitstensor core, n.: A small, specialized part of an AI chip that does one job — multiplying small blocks of numbers together — far faster than ordinary circuits can. and energy. In AI engineering it explains why prefillprefill, n.: The first stage of an AI's reply, in which it reads your whole prompt in one go before it starts writing. Long prompts make this stage take longer. is compute-bound while decodedecode phase, n.: The second stage of an AI's reply, in which it writes the answer one word-piece at a time, each step depending on everything written so far. is memory-bound, and guides choices of batch size, quantizationquantization, n.: Storing an AI model's numbers with fewer digits, like rounding prices to the nearest dollar, so the model takes less memory and runs faster, usually with only a small loss of accuracy. and parallelism.
Origin
Introduced by Samuel Williams, Andrew Waterman and David Patterson of the University of California, Berkeley, in 'Roofline: An Insightful Visual Performance Model for Multicore Architectures' (Communications of the ACM, April 2009), building on Williams's 2008 doctoral work.
Related
- arithmetic intensityarithmetic intensity, n.: How much calculating a task does for each piece of data it fetches from memory. Tasks with little calculation per fetch are slowed by memory, not by the chip's math speed.
- memory bandwidthmemory bandwidth, n.: How fast data can be moved between a chip's memory and its calculators. For chatbots, this often decides how quickly words appear, more than raw calculating speed.
- FLOPSFLOPS, abbr.: Floating-point operations per second: how many arithmetic steps a computer can do each second. It is the standard speed rating for supercomputers and AI chips.
- memory wallmemory wall, n.: The growing gap between how fast chips can calculate and how fast they can fetch data from memory. Processors increasingly sit idle, waiting for numbers to arrive.
- prefillprefill, n.: The first stage of an AI's reply, in which it reads your whole prompt in one go before it starts writing. Long prompts make this stage take longer.
- decode phasedecode phase, n.: The second stage of an AI's reply, in which it writes the answer one word-piece at a time, each step depending on everything written so far.
Sources
- Williams, S., Waterman, A. & Patterson, D. (2009). Roofline: An Insightful Visual Performance Model for Multicore Architectures. Communications of the ACM 52(4), 65–76. doi:10.1145/1498765.1498785.
Mentioned in
FLOPSFLOPS, abbr.: Floating-point operations per second: how many arithmetic steps a computer can do each second. It is the standard speed rating for supercomputers and AI chips., memory bandwidthmemory bandwidth, n.: How fast data can be moved between a chip's memory and its calculators. For chatbots, this often decides how quickly words appear, more than raw calculating speed., arithmetic intensityarithmetic intensity, n.: How much calculating a task does for each piece of data it fetches from memory. Tasks with little calculation per fetch are slowed by memory, not by the chip's math speed., memory wallmemory wall, n.: The growing gap between how fast chips can calculate and how fast they can fetch data from memory. Processors increasingly sit idle, waiting for numbers to arrive., memory-boundmemory-bound, adj.: Describes a computer task slowed by how fast data can be fetched from memory rather than by how fast the chip can calculate — like a fast cook kept waiting for deliveries., model FLOPs utilizationmodel FLOPs utilization, n.: The share of a computer's top calculating speed that actually goes into useful AI training work. Real systems often reach only around a third to a half.