mixed-precision training
In plain wordsTraining a model with shorter, less exact numbers for most of the arithmetic and full-length numbers only where accuracy really matters, which saves a great deal of memory and lets modern chips train far faster.
A training technique that performs most arithmetic — the matrix multiplications of the forward passforward pass, n.: The step where a neural network takes an input and runs it through its layers, one after another, to produce an output — its answer or prediction. and backward passbackward pass, n.: The step in training where the network works backward from its error, layer by layer, to calculate how each internal setting should change., and stored activations — in low-precision floating-point formatsfloating-point format, n.: A way of storing numbers in a computer like scientific notation — a few digits plus a power of two — trading accuracy for range. AI uses ever-smaller formats to compute faster. such as FP16FP16, abbr.: A 16-bit way of storing numbers that uses half the memory of the standard 32-bit format. It is fast for AI but can't represent very large or very tiny values., bfloat16bfloat16, n.: A 16-bit number format invented at Google for AI. It keeps the huge range of the standard 32-bit format but drops precision, which neural networks can usually tolerate. or FP8FP8, abbr.: An 8-bit number format that uses a quarter of the memory of the standard 32-bit format, letting new AI chips train and run models roughly twice as fast as with 16 bits., while keeping numerically sensitive parts in FP32FP32, abbr.: The standard 32-bit way computers store numbers with fractional parts, keeping about seven significant digits. It was the default for training AI until faster, smaller formats took over.: a master copy of the weights to which small updates are added, the optimizeroptimizer, n.: The rule a training program uses to turn measured errors into actual changes to a model's internal numbers, deciding the direction and size of each adjustment. state, large reductions such as sums and softmaxsoftmax, n.: A formula that turns a list of raw scores into probabilities — all positive and adding up to one — with the biggest share going to the highest score. normalizers, and the loss. Low-precision operands cut memory traffic and run on the much faster tensor corestensor core, n.: A small, specialized part of an AI chip that does one job — multiplying small blocks of numbers together — far faster than ordinary circuits can. of modern accelerators. FP16's narrow exponent range lets small gradients underflow to zero, so FP16 training multiplies the loss by a large factor before backpropagation and divides it out afterwards (loss scaling, usually adjusted dynamically); bfloat16 keeps FP32's range and normally needs no scaling, which is why it became the default for large models. FP8 training adds per-tensor or finer-grained scaling factors and has been used at scale, for example for DeepSeek-V3 (2024).
With bf16 mixed precision the run fit on half the GPUs and needed no loss scaling.
Origin
Paulius Micikevicius and colleagues at NVIDIA and Baidu set out the standard recipe — FP16 arithmetic, an FP32 master copy of weights and loss scaling — in 'Mixed Precision Training' (arXiv October 2017; ICLR 2018), alongside NVIDIA's Volta tensor cores. Google's bfloat16 format on TPUs and FP8 support in NVIDIA's Hopper GPUs (2022) extended the approach.
Usage note
Distinct from quantizationquantization, n.: Storing an AI model's numbers with fewer digits, like rounding prices to the nearest dollar, so the model takes less memory and runs faster, usually with only a small loss of accuracy., which lowers precision for inference, and from quantization-aware trainingquantization-aware training, n.: Training or fine-tuning an AI model while simulating the low-precision numbers it will use later, so that it learns to work well despite the rounding.. 'Precision' here means numeric precision, not the evaluation metric precisionprecision, n.: Of all the items a model flagged as positive, the share that really were positive; a spam filter with high precision rarely sends genuine emails to the spam folder..
Related
- bfloat16bfloat16, n.: A 16-bit number format invented at Google for AI. It keeps the huge range of the standard 32-bit format but drops precision, which neural networks can usually tolerate.
- FP16FP16, abbr.: A 16-bit way of storing numbers that uses half the memory of the standard 32-bit format. It is fast for AI but can't represent very large or very tiny values.
- FP8FP8, abbr.: An 8-bit number format that uses a quarter of the memory of the standard 32-bit format, letting new AI chips train and run models roughly twice as fast as with 16 bits.
- floating-point formatfloating-point format, n.: A way of storing numbers in a computer like scientific notation — a few digits plus a power of two — trading accuracy for range. AI uses ever-smaller formats to compute faster.
- quantizationquantization, n.: Storing an AI model's numbers with fewer digits, like rounding prices to the nearest dollar, so the model takes less memory and runs faster, usually with only a small loss of accuracy.
- tensor coretensor core, n.: A small, specialized part of an AI chip that does one job — multiplying small blocks of numbers together — far faster than ordinary circuits can.
- numerical stabilitynumerical stability, n.: Whether a computer calculation stays accurate despite the tiny rounding errors in every step, instead of blowing up into nonsense such as infinities.
Sources
- Micikevicius, P. et al. (2018). Mixed Precision Training. ICLR 2018. arxiv.org/abs/1710.03740
Mentioned in
bfloat16bfloat16, n.: A 16-bit number format invented at Google for AI. It keeps the huge range of the standard 32-bit format but drops precision, which neural networks can usually tolerate., floating-point formatfloating-point format, n.: A way of storing numbers in a computer like scientific notation — a few digits plus a power of two — trading accuracy for range. AI uses ever-smaller formats to compute faster., FP16FP16, abbr.: A 16-bit way of storing numbers that uses half the memory of the standard 32-bit format. It is fast for AI but can't represent very large or very tiny values., FP32FP32, abbr.: The standard 32-bit way computers store numbers with fractional parts, keeping about seven significant digits. It was the default for training AI until faster, smaller formats took over., FP8FP8, abbr.: An 8-bit number format that uses a quarter of the memory of the standard 32-bit format, letting new AI chips train and run models roughly twice as fast as with 16 bits., full fine-tuningfull fine-tuning, n.: Fine-tuning in which every one of a model's internal numbers is allowed to change — the most flexible option, but also the most expensive in memory and storage., gradient checkpointinggradient checkpointing, n.: A memory-saving trick: instead of storing every intermediate result needed for learning, the computer keeps only a few and recalculates the rest when needed, trading extra computing time for memory., numerical stabilitynumerical stability, n.: Whether a computer calculation stays accurate despite the tiny rounding errors in every step, instead of blowing up into nonsense such as infinities., precisionprecision, n.: Of all the items a model flagged as positive, the share that really were positive; a spam filter with high precision rarely sends genuine emails to the spam folder., tensor coretensor core, n.: A small, specialized part of an AI chip that does one job — multiplying small blocks of numbers together — far faster than ordinary circuits can., training instabilitytraining instability, n.: The tendency of very large training runs to go wrong — the error suddenly jumping or blowing up — which can waste weeks of expensive computing unless engineers prevent or repair it., AdafactorAdafactor, prop. n.: A memory-saving cousin of the popular Adam training method. Instead of keeping extra statistics for every single weight, it keeps summaries for each row and column of a weight table., DeepSpeedDeepSpeed, prop. n.: Free software from Microsoft for training and running AI models too big for ordinary tools, spread over many chips; it is best known for a memory-saving technique called ZeRO., quantization-aware trainingquantization-aware training, n.: Training or fine-tuning an AI model while simulating the low-precision numbers it will use later, so that it learns to work well despite the rounding., ZeROZeRO, prop. n.: A memory-saving technique for training big models on many chips: instead of every chip keeping its own complete copy of all the training bookkeeping, the chips divide it among themselves and share pieces only when needed.