Huang's law
In plain wordsA claim, named after Nvidia's chief executive, that AI chip performance improves much faster than Moore's law because chips, software and number formats all improve together.
An informal rule of thumb that the throughput of GPUsGPU, abbr.: A computer chip first designed to draw video-game graphics that turned out to be ideal for AI, because it can do thousands of simple calculations at the same time. on AI workloads grows by more than a factor of two every two years, outpacing what added transistors alone provide, because gains compound across chip architecture, interconnectsinterconnect, n.: The wiring and networking that connect chips to each other — inside a server, across a data center, or within a single chip — so they can share data and work as one., software and, above all, lower-precision number formats. Supporters point to per-chip AI throughput rising by orders of magnitude within a decade, much of it from moving between number formats (FP32FP32, abbr.: The standard 32-bit way computers store numbers with fractional parts, keeping about seven significant digits. It was the default for training AI until faster, smaller formats took over. to FP16FP16, abbr.: A 16-bit way of storing numbers that uses half the memory of the standard 32-bit format. It is fast for AI but can't represent very large or very tiny values. to FP8FP8, abbr.: An 8-bit number format that uses a quarter of the memory of the standard 32-bit format, letting new AI chips train and run models roughly twice as fast as with 16 bits. to FP4FP4, abbr.: A tiny 4-bit number format that can represent only sixteen different values. With clever scaling tricks it lets the newest AI chips run, and even train, models faster and with less memory.) and adding structured sparsitysparsity, n.: The property of being mostly zeros or mostly switched off. AI systems exploit it by skipping the empty parts, so they need less memory and computation.. Critics reply that such comparisons mix precisions, that format gains cannot repeat indefinitely, and that broader measures grow more slowly: Epoch AI (2022) found GPU price-performance in FLOPS per dollar doubling only about every 2.5 years between 2006 and 2021.
Origin
Named after Nvidia co-founder and chief executive Jensen Huang, who at Nvidia's GTC conference in March 2018 said its GPUs were 25 times faster than five years earlier. Tekla S. Perry used the name in the title of an IEEE Spectrum article in April 2018, noting that Huang had not coined it himself; Christopher Mims's Wall Street Journal column of September 2020 popularized it.
Related
- Moore's lawMoore's law, n.: Gordon Moore's prediction that chipmakers could pack twice as many transistors onto a chip roughly every two years — a pace that for decades made computers steadily faster and cheaper, and has slowed but not stopped.
- GPUGPU, abbr.: A computer chip first designed to draw video-game graphics that turned out to be ideal for AI, because it can do thousands of simple calculations at the same time.
- floating-point formatfloating-point format, n.: A way of storing numbers in a computer like scientific notation — a few digits plus a power of two — trading accuracy for range. AI uses ever-smaller formats to compute faster.
- hardware lotteryhardware lottery, n.: The idea that research ideas succeed partly by luck — because they happen to suit the computer chips available at the time — rather than purely because they are better.
- AI acceleratorAI accelerator, n.: Any chip built or tuned to speed up AI — graphics chips, Google's TPUs, phone 'neural engines' and many startup designs. Such chips do the heavy math of training and running models.
- Moore's lawMoore's law, n.: Gordon Moore's prediction that chipmakers could pack twice as many transistors onto a chip roughly every two years — a pace that for decades made computers steadily faster and cheaper, and has slowed but not stopped.
Sources
- Perry, T. S. (2018). Move Over, Moore's Law: Make Way for Huang's Law. IEEE Spectrum. spectrum.ieee.org/move-over-moores-law-make-way-for-huangs-law
- Mims, C. (2020). Huang's Law Is the New Moore's Law, and Explains Why Nvidia Wants Arm. The Wall Street Journal, 19 September 2020.
Mentioned in
Moore's lawMoore's law, n.: Gordon Moore's prediction that chipmakers could pack twice as many transistors onto a chip roughly every two years — a pace that for decades made computers steadily faster and cheaper, and has slowed but not stopped., hardware lotteryhardware lottery, n.: The idea that research ideas succeed partly by luck — because they happen to suit the computer chips available at the time — rather than purely because they are better.