microscaling formats
In plain wordsNumber formats that store small groups of numbers with one shared scale factor, so each number can use very few bits while the group as a whole stays accurate.
A family of block-scaled number formats in which a block of k elements (32 in the standard) stored in a narrow type shares one 8-bit power-of-two scale (E8M0). The Open Compute Project's MX specification defines MXFP8 (E4M3 or E5M2 elements), MXFP6 (E3M2, E2M3), MXFP4 (E2M1) and MXINT8. Because the scale tracks the local magnitude of each small block, an outlier in one block does not destroy precision elsewhere, letting 4- to 8-bit elements approach the accuracy of 16-bit formats with modest overhead (an FP4FP4, abbr.: A tiny 4-bit number format that can represent only sixteen different values. With clever scaling tricks it lets the newest AI chips run, and even train, models faster and with less memory. block of 32 costs 4.25 bits per value). Matrix unitstensor core, n.: A small, specialized part of an AI chip that does one job — multiplying small blocks of numbers together — far faster than ordinary circuits can. in Nvidia Blackwell, AMD MI350 and other accelerators apply the scales in hardware. Variants with smaller blocks and finer scales, such as Nvidia's NVFP4 (16-element blocks with FP8FP8, abbr.: An 8-bit number format that uses a quarter of the memory of the standard 32-bit format, letting new AI chips train and run models roughly twice as fast as with 16 bits. scales), trade a little storage for accuracy.
Origin
Grew out of block floating point, long used in signal processing, and Microsoft's MSFP formats for its Brainwave FPGA service (c. 2018–2020). The OCP Microscaling Formats v1.0 specification was released in September 2023 by AMD, Arm, Intel, Meta, Microsoft, Nvidia and Qualcomm; Bita Darvish Rouhani and colleagues' accompanying paper (October 2023) reported sub-8-bit training of generative language models.
Related
- FP4FP4, abbr.: A tiny 4-bit number format that can represent only sixteen different values. With clever scaling tricks it lets the newest AI chips run, and even train, models faster and with less memory.
- FP8FP8, abbr.: An 8-bit number format that uses a quarter of the memory of the standard 32-bit format, letting new AI chips train and run models roughly twice as fast as with 16 bits.
- floating-point formatfloating-point format, n.: A way of storing numbers in a computer like scientific notation — a few digits plus a power of two — trading accuracy for range. AI uses ever-smaller formats to compute faster.
- quantizationquantization, n.: Storing an AI model's numbers with fewer digits, like rounding prices to the nearest dollar, so the model takes less memory and runs faster, usually with only a small loss of accuracy.
- INT8INT8, abbr.: Storing numbers as whole numbers from −128 to 127 using 8 bits. AI models converted to this format run faster and need a quarter of the memory of 32-bit versions.
- tensor coretensor core, n.: A small, specialized part of an AI chip that does one job — multiplying small blocks of numbers together — far faster than ordinary circuits can.
Sources
- Open Compute Project (2023). OCP Microscaling Formats (MX) Specification, Version 1.0. opencompute.org/documents/ocp-microscaling-formats-mx-v1-0-spec-final-pdf
- Rouhani, B. D. et al. (2023). Microscaling Data Formats for Deep Learning. arXiv:2310.10537. arxiv.org/abs/2310.10537
Mentioned in
floating-point formatfloating-point format, n.: A way of storing numbers in a computer like scientific notation — a few digits plus a power of two — trading accuracy for range. AI uses ever-smaller formats to compute faster., FP8FP8, abbr.: An 8-bit number format that uses a quarter of the memory of the standard 32-bit format, letting new AI chips train and run models roughly twice as fast as with 16 bits., tensor coretensor core, n.: A small, specialized part of an AI chip that does one job — multiplying small blocks of numbers together — far faster than ordinary circuits can., FP4FP4, abbr.: A tiny 4-bit number format that can represent only sixteen different values. With clever scaling tricks it lets the newest AI chips run, and even train, models faster and with less memory.