FP8
In plain wordsAn 8-bit number format that uses a quarter of the memory of the standard 32-bit format, letting new AI chips train and run models roughly twice as fast as with 16 bits.
A family of 8-bit floating-point formatsfloating-point format, n.: A way of storing numbers in a computer like scientific notation — a few digits plus a power of two — trading accuracy for range. AI uses ever-smaller formats to compute faster. for deep learning, chiefly E4M3 (4 exponent and 3 significand bits; maximum 448, no infinities) and E5M2 (5 and 2; maximum 57,344, with IEEE-style infinities and NaNs). A common recipe uses E4M3 for weights and activations and the wider-range E5M2 for gradients, with per-tensor or finer-grained scale factors that keep values in range and higher-precision accumulation; Nvidia's Transformer Engine software automates the scaling. Hardware support arrived in 2022, most prominently in Nvidia's H100H100, prop. n.: Nvidia's flagship AI chip of 2022–2024, so scarce and sought-after during the generative-AI boom that it became a unit of measure for AI computing power., and spread to AMD, Intel, Google and other accelerators, doubling tensor coretensor core, n.: A small, specialized part of an AI chip that does one job — multiplying small blocks of numbers together — far faster than ordinary circuits can. throughput over 16-bit formats. FP8 is widely used for inferenceinference, n.: Drawing a conclusion from evidence. In today's AI it most often means running an already-trained model to get an answer, as opposed to training it. and increasingly for training — DeepSeek-V3 (2024) was trained largely in FP8 with fine-grained block scaling — and a block-scaled variant, MXFP8, is part of the microscaling formatsmicroscaling formats, n.: Number formats that store small groups of numbers with one shared scale factor, so each number can use very few bits while the group as a whole stays accurate. specification.
Origin
Proposed as a common interchange format by Paulius Micikevicius and colleagues from Nvidia, Arm and Intel in 'FP8 Formats for Deep Learning' (September 2022), which reported FP8 training matching 16-bit quality on models of up to 175 billion parameters. IBM researchers had demonstrated 8-bit floating-point training in 2018–2019.
Related
- floating-point formatfloating-point format, n.: A way of storing numbers in a computer like scientific notation — a few digits plus a power of two — trading accuracy for range. AI uses ever-smaller formats to compute faster.
- FP4FP4, abbr.: A tiny 4-bit number format that can represent only sixteen different values. With clever scaling tricks it lets the newest AI chips run, and even train, models faster and with less memory.
- bfloat16bfloat16, n.: A 16-bit number format invented at Google for AI. It keeps the huge range of the standard 32-bit format but drops precision, which neural networks can usually tolerate.
- microscaling formatsmicroscaling formats, n.: Number formats that store small groups of numbers with one shared scale factor, so each number can use very few bits while the group as a whole stays accurate.
- quantizationquantization, n.: Storing an AI model's numbers with fewer digits, like rounding prices to the nearest dollar, so the model takes less memory and runs faster, usually with only a small loss of accuracy.
- mixed-precision trainingmixed-precision training, n.: Training a model with shorter, less exact numbers for most of the arithmetic and full-length numbers only where accuracy really matters, which saves a great deal of memory and lets modern chips train far faster.
- H100H100, prop. n.: Nvidia's flagship AI chip of 2022–2024, so scarce and sought-after during the generative-AI boom that it became a unit of measure for AI computing power.
Sources
- Micikevicius, P. et al. (2022). FP8 Formats for Deep Learning. arXiv:2209.05433. arxiv.org/abs/2209.05433
Mentioned in
AI acceleratorAI accelerator, n.: Any chip built or tuned to speed up AI — graphics chips, Google's TPUs, phone 'neural engines' and many startup designs. Such chips do the heavy math of training and running models., bfloat16bfloat16, n.: A 16-bit number format invented at Google for AI. It keeps the huge range of the standard 32-bit format but drops precision, which neural networks can usually tolerate., FlashAttentionFlashAttention, prop. n.: A faster way for AI chips to compute attention, the step in which a language model relates every word to every other. It gets exactly the same answer while moving far less data around the chip., floating-point formatfloating-point format, n.: A way of storing numbers in a computer like scientific notation — a few digits plus a power of two — trading accuracy for range. AI uses ever-smaller formats to compute faster., FLOPSFLOPS, abbr.: Floating-point operations per second: how many arithmetic steps a computer can do each second. It is the standard speed rating for supercomputers and AI chips., FP32FP32, abbr.: The standard 32-bit way computers store numbers with fractional parts, keeping about seven significant digits. It was the default for training AI until faster, smaller formats took over., H100H100, prop. n.: Nvidia's flagship AI chip of 2022–2024, so scarce and sought-after during the generative-AI boom that it became a unit of measure for AI computing power., INT8INT8, abbr.: Storing numbers as whole numbers from −128 to 127 using 8 bits. AI models converted to this format run faster and need a quarter of the memory of 32-bit versions., mixed-precision trainingmixed-precision training, n.: Training a model with shorter, less exact numbers for most of the arithmetic and full-length numbers only where accuracy really matters, which saves a great deal of memory and lets modern chips train far faster., numerical stabilitynumerical stability, n.: Whether a computer calculation stays accurate despite the tiny rounding errors in every step, instead of blowing up into nonsense such as infinities., quantizationquantization, n.: Storing an AI model's numbers with fewer digits, like rounding prices to the nearest dollar, so the model takes less memory and runs faster, usually with only a small loss of accuracy., tensor coretensor core, n.: A small, specialized part of an AI chip that does one job — multiplying small blocks of numbers together — far faster than ordinary circuits can., training instabilitytraining instability, n.: The tendency of very large training runs to go wrong — the error suddenly jumping or blowing up — which can waste weeks of expensive computing unless engineers prevent or repair it., vLLMvLLM, prop. n.: A free, open-source program for running large language models on servers efficiently, so one machine can answer many more users at once. Many companies use it to host AI models., FP4FP4, abbr.: A tiny 4-bit number format that can represent only sixteen different values. With clever scaling tricks it lets the newest AI chips run, and even train, models faster and with less memory., Huang's lawHuang's law, n.: A claim, named after Nvidia's chief executive, that AI chip performance improves much faster than Moore's law because chips, software and number formats all improve together., microscaling formatsmicroscaling formats, n.: Number formats that store small groups of numbers with one shared scale factor, so each number can use very few bits while the group as a whole stays accurate., TensorRTTensorRT, prop. n.: NVIDIA's software for making a trained neural network run as fast as possible on NVIDIA graphics chips, by merging and simplifying its calculations and testing which way of doing each one is quickest., TensorRT-LLMTensorRT-LLM, prop. n.: NVIDIA's free software library for running large language models as fast as possible on NVIDIA's own graphics chips, used by many companies that host AI models.