high-bandwidth memory
In plain wordsVery fast memory made by stacking memory chips on top of each other and placing them right next to the processor, so AI chips can be fed data quickly enough.
A type of DRAMDRAM, abbr.: The main working memory of computers and AI chips: dense and fairly cheap, but slower than on-chip memory and needing constant refreshing to keep its data. in which 4 to 16 memory dies are stacked and connected vertically by through-silicon vias, then placed beside a processor on a silicon interposer (2.5D packagingadvanced packaging, n.: Ways of joining several chips very closely inside one package — side by side on a silicon base or stacked on top of each other — so they act almost like one bigger chip.) and linked by a very wide interface — 1,024 bits per stack through HBM3E and 2,048 bits in HBM4 — instead of the narrow, fast buses of conventional DRAM. Per-stack bandwidth has risen from 128 GB/s (HBM1) to about 1 TB/s (HBM3E) and up to 2 TB/s (HBM4). Accelerators carry several stacks: the H100H100, prop. n.: Nvidia's flagship AI chip of 2022–2024, so scarce and sought-after during the generative-AI boom that it became a unit of measure for AI computing power. has 80 GB at 3.35 TB/s and Blackwell-generation GPUs around 192 GB or more. Because model weightsweights, n.: The learned numbers inside a neural network that set how strongly each part influences the next. The finished set of weights is, in effect, the trained model itself. and KV cachesKV cache, n.: A memory store where a chatbot keeps its working notes on everything already read and written, so each new word does not require re-reading the whole conversation from scratch. stream from HBM during inferenceinference, n.: Drawing a conclusion from evidence. In today's AI it most often means running an already-trained model to get an answer, as opposed to training it., its capacity and bandwidthmemory bandwidth, n.: How fast data can be moved between a chip's memory and its calculators. For chatbots, this often decides how quickly words appear, more than raw calculating speed. often limit speed more than arithmetic does. HBM is costly to make and is produced by only three firms — SK hynix, Samsung and Micron — making it a recurring bottleneck in AI chip supply.
Origin
Developed by AMD and SK hynix from 2008 and standardized by JEDEC as JESD235 in October 2013. It first shipped in AMD's Fiji GPU (Radeon R9 Fury X, June 2015) and entered AI hardware with Nvidia's Tesla P100 (2016). JEDEC published HBM3 in January 2022 and HBM4 in April 2025.
Usage note
Producing a bit of HBM takes roughly three times the wafer capacity of standard DDR5, according to Micron, so AI demand has squeezed commodity DRAM supply and pushed up its prices since 2025.
Related
- memory bandwidthmemory bandwidth, n.: How fast data can be moved between a chip's memory and its calculators. For chatbots, this often decides how quickly words appear, more than raw calculating speed.
- memory wallmemory wall, n.: The growing gap between how fast chips can calculate and how fast they can fetch data from memory. Processors increasingly sit idle, waiting for numbers to arrive.
- DRAMDRAM, abbr.: The main working memory of computers and AI chips: dense and fairly cheap, but slower than on-chip memory and needing constant refreshing to keep its data.
- SRAMSRAM, abbr.: The fastest kind of computer memory, built directly into processor chips. It is quick but takes up a lot of space, so chips hold only a little of it.
- advanced packagingadvanced packaging, n.: Ways of joining several chips very closely inside one package — side by side on a silicon base or stacked on top of each other — so they act almost like one bigger chip.
- VRAMVRAM, abbr.: The memory on a graphics card. For running AI at home it is the key number: the model has to fit in it, so more memory means bigger models.
- KV cacheKV cache, n.: A memory store where a chatbot keeps its working notes on everything already read and written, so each new word does not require re-reading the whole conversation from scratch.
- SRAMSRAM, abbr.: The fastest kind of computer memory, built directly into processor chips. It is quick but takes up a lot of space, so chips hold only a little of it.
Mentioned in
AI acceleratorAI accelerator, n.: Any chip built or tuned to speed up AI — graphics chips, Google's TPUs, phone 'neural engines' and many startup designs. Such chips do the heavy math of training and running models., export controls on AI chipsexport controls on AI chips, n.: Government rules that limit which countries and companies may buy the most advanced AI chips and the machines used to make them. The United States uses them mainly to slow China's progress in AI., GPUGPU, abbr.: A computer chip first designed to draw video-game graphics that turned out to be ideal for AI, because it can do thousands of simple calculations at the same time., AMDAMD, prop. n.: An American chip company that makes computer processors and graphics chips, and Nvidia's main rival in the chips used for artificial intelligence., FlashAttentionFlashAttention, prop. n.: A faster way for AI chips to compute attention, the step in which a language model relates every word to every other. It gets exactly the same answer while moving far less data around the chip., H100H100, prop. n.: Nvidia's flagship AI chip of 2022–2024, so scarce and sought-after during the generative-AI boom that it became a unit of measure for AI computing power., memory bandwidthmemory bandwidth, n.: How fast data can be moved between a chip's memory and its calculators. For chatbots, this often decides how quickly words appear, more than raw calculating speed., TPUTPU, abbr.: Google's own family of chips built specifically for AI, used to train and run its Gemini models and rented to other companies through its cloud., TSMCTSMC, prop. n.: The Taiwanese company that manufactures most of the world's advanced computer chips, including nearly all of the specialized chips used to train and run AI., VRAMVRAM, abbr.: The memory on a graphics card. For running AI at home it is the key number: the model has to fit in it, so more memory means bigger models., advanced packagingadvanced packaging, n.: Ways of joining several chips very closely inside one package — side by side on a silicon base or stacked on top of each other — so they act almost like one bigger chip., chipletchiplet, n.: One of several smaller chips that are made separately and then joined inside a single package to work as one processor, which is cheaper than making one giant chip and lets makers reuse parts., DRAMDRAM, abbr.: The main working memory of computers and AI chips: dense and fairly cheap, but slower than on-chip memory and needing constant refreshing to keep its data., memory wallmemory wall, n.: The growing gap between how fast chips can calculate and how fast they can fetch data from memory. Processors increasingly sit idle, waiting for numbers to arrive., memory-boundmemory-bound, adj.: Describes a computer task slowed by how fast data can be fetched from memory rather than by how fast the chip can calculate — like a fast cook kept waiting for deliveries., offloadingoffloading, n.: Moving part of an AI model's data or work out of a chip's scarce fast memory onto ordinary memory, a disk or another computer, so that models too big for the chip can still run., rack-scale systemrack-scale system, n.: An AI computer designed as one whole cabinet of chips, cooling and wiring, rather than separate servers, so all the chips inside can work together like one big chip., SRAMSRAM, abbr.: The fastest kind of computer memory, built directly into processor chips. It is quick but takes up a lot of space, so chips hold only a little of it., von Neumann bottleneckvon Neumann bottleneck, n.: The traffic jam caused by ordinary computers keeping data and processing in separate places, so information must constantly shuttle back and forth through one narrow channel.