bfloat16
In plain wordsA 16-bit number format invented at Google for AI. It keeps the huge range of the standard 32-bit format but drops precision, which neural networks can usually tolerate.
A 16-bit floating-point formatfloating-point format, n.: A way of storing numbers in a computer like scientific notation — a few digits plus a power of two — trading accuracy for range. AI uses ever-smaller formats to compute faster. with 1 sign bit, 8 exponent bits and 7 stored significand bits — in effect the upper half of an FP32FP32, abbr.: The standard 32-bit way computers store numbers with fractional parts, keeping about seven significant digits. It was the default for training AI until faster, smaller formats took over. number. Sharing FP32's exponent gives it the same dynamic range (roughly 10^-38 to 3 × 10^38), so gradients rarely underflow or overflow and training usually needs no loss scaling; the price is coarse precision (2–3 significant decimal digits), mitigated by accumulating sums in FP32. Conversion to and from FP32 is a simple truncation or rounding. Supported by Google TPUsTPU, abbr.: Google's own family of chips built specifically for AI, used to train and run its Gemini models and rented to other companies through its cloud. from v2, recent Intel and AMD CPUs, Nvidia GPUs from the A100 (2020), AMD accelerators and Arm processors, bfloat16 became the default format for training large language modelslarge language model, n.: A computer program trained on enormous amounts of text to predict the words that come next. At vast scale, that simple skill lets it chat, answer questions, translate, summarize and write code. in the early 2020s, and model checkpoints are commonly distributed in it.
Train in bfloat16 and you can usually drop the loss-scaling code that FP16 needed.
Origin
Developed by Google Brain ('b' for 'brain') and introduced with the second-generation TPU in 2017. Dhiraj Kalamkar and colleagues at Intel and Facebook published a broad study in 2019 showing that it matched FP32 training results across domains without changing hyperparameters.
Usage note
Not an IEEE 754 format. With only 8 significant bits, adding a small update to a large weight can leave it unchanged, one reason optimizers keep FP32 master copies.
Related
- floating-point formatfloating-point format, n.: A way of storing numbers in a computer like scientific notation — a few digits plus a power of two — trading accuracy for range. AI uses ever-smaller formats to compute faster.
- FP32FP32, abbr.: The standard 32-bit way computers store numbers with fractional parts, keeping about seven significant digits. It was the default for training AI until faster, smaller formats took over.
- FP16FP16, abbr.: A 16-bit way of storing numbers that uses half the memory of the standard 32-bit format. It is fast for AI but can't represent very large or very tiny values.
- FP8FP8, abbr.: An 8-bit number format that uses a quarter of the memory of the standard 32-bit format, letting new AI chips train and run models roughly twice as fast as with 16 bits.
- TPUTPU, abbr.: Google's own family of chips built specifically for AI, used to train and run its Gemini models and rented to other companies through its cloud.
- mixed-precision trainingmixed-precision training, n.: Training a model with shorter, less exact numbers for most of the arithmetic and full-length numbers only where accuracy really matters, which saves a great deal of memory and lets modern chips train far faster.
- FP16FP16, abbr.: A 16-bit way of storing numbers that uses half the memory of the standard 32-bit format. It is fast for AI but can't represent very large or very tiny values.
Sources
- Kalamkar, D. et al. (2019). A Study of BFLOAT16 for Deep Learning Training. arXiv:1905.12322. arxiv.org/abs/1905.12322
Mentioned in
AI acceleratorAI accelerator, n.: Any chip built or tuned to speed up AI — graphics chips, Google's TPUs, phone 'neural engines' and many startup designs. Such chips do the heavy math of training and running models., floating-point formatfloating-point format, n.: A way of storing numbers in a computer like scientific notation — a few digits plus a power of two — trading accuracy for range. AI uses ever-smaller formats to compute faster., FLOPSFLOPS, abbr.: Floating-point operations per second: how many arithmetic steps a computer can do each second. It is the standard speed rating for supercomputers and AI chips., FP16FP16, abbr.: A 16-bit way of storing numbers that uses half the memory of the standard 32-bit format. It is fast for AI but can't represent very large or very tiny values., FP32FP32, abbr.: The standard 32-bit way computers store numbers with fractional parts, keeping about seven significant digits. It was the default for training AI until faster, smaller formats took over., FP8FP8, abbr.: An 8-bit number format that uses a quarter of the memory of the standard 32-bit format, letting new AI chips train and run models roughly twice as fast as with 16 bits., H100H100, prop. n.: Nvidia's flagship AI chip of 2022–2024, so scarce and sought-after during the generative-AI boom that it became a unit of measure for AI computing power., mixed-precision trainingmixed-precision training, n.: Training a model with shorter, less exact numbers for most of the arithmetic and full-length numbers only where accuracy really matters, which saves a great deal of memory and lets modern chips train far faster., numerical stabilitynumerical stability, n.: Whether a computer calculation stays accurate despite the tiny rounding errors in every step, instead of blowing up into nonsense such as infinities., precisionprecision, n.: Of all the items a model flagged as positive, the share that really were positive; a spam filter with high precision rarely sends genuine emails to the spam folder., tensor coretensor core, n.: A small, specialized part of an AI chip that does one job — multiplying small blocks of numbers together — far faster than ordinary circuits can., TPUTPU, abbr.: Google's own family of chips built specifically for AI, used to train and run its Gemini models and rented to other companies through its cloud., training instabilitytraining instability, n.: The tendency of very large training runs to go wrong — the error suddenly jumping or blowing up — which can waste weeks of expensive computing unless engineers prevent or repair it.