VRAM

AI usageinformal

The dedicated memory attached to a GPUGPU, abbr.: A computer chip first designed to draw video-game graphics that turned out to be ideal for AI, because it can do thousands of simple calculations at the same time. — GDDR on consumer cards, HBMhigh-bandwidth memory, n.: Very fast memory made by stacking memory chips on top of each other and placing them right next to the processor, so AI chips can be fed data quickly enough. on data-center accelerators — and by extension its capacity, usually the first constraint on which models a given GPU can train or run. Serving a model needs room for its weightsweights, n.: The learned numbers inside a neural network that set how strongly each part influences the next. The finished set of weights is, in effect, the trained model itself. (about 2 bytes per parameterparameter, n.: One of the adjustable numbers inside an AI model that are tuned during training. Large language models have billions of them, and together they store what the model has learned. in 16-bit precision, 0.5 bytes at 4-bit), for the KV cacheKV cache, n.: A memory store where a chatbot keeps its working notes on everything already read and written, so each new word does not require re-reading the whole conversation from scratch., which grows with context length and batch size, and for runtime overhead; training needs several times more for gradientsgradient, n.: A list of slopes showing which way, and how steeply, a quantity rises as each of its inputs changes. Training nudges an AI's settings in the opposite direction, downhill., optimizer states and activations. Hobbyists running open-weight modelsopen-weight model, n.: An AI model whose trained internal numbers are published for anyone to download and run on their own computers, even if the data and code used to make it stay private. locally therefore rely on quantizationquantization, n.: Storing an AI model's numbers with fewer digits, like rounding prices to the nearest dollar, so the model takes less memory and runs faster, usually with only a small loss of accuracy. and offloadingoffloading, n.: Moving part of an AI model's data or work out of a chip's scarce fast memory onto ordinary memory, a disk or another computer, so that models too big for the chip can still run. to system memory, and consumer cards with 24–32 GB are prized.

A 70B model at 4-bit needs about 40 GB of VRAM, so it won't fit on one 24 GB card.

Original sensehistorical

Video RAM in the original hardware sense: a DRAMDRAM, abbr.: The main working memory of computers and AI chips: dense and fairly cheap, but slower than on-chip memory and needing constant refreshing to keep its data. variant with two independent access paths, one through which the processor updated the image and one that streamed pixels continuously to the screen, used to hold the displayed frame in workstations and PCs from the mid-1980s into the 1990s.