H100
In plain wordsNvidia's flagship AI chip of 2022–2024, so scarce and sought-after during the generative-AI boom that it became a unit of measure for AI computing power.
A data-center GPUGPU, abbr.: A computer chip first designed to draw video-game graphics that turned out to be ideal for AI, because it can do thousands of simple calculations at the same time. of Nvidia's Hopper architecture (named for Grace Hopper), announced in March 2022 and shipping from late 2022. Built on TSMC's 4N processprocess node, n.: The label for a generation of chip-making technology, like '3 nanometer'. Smaller numbers mean newer, denser chips, but the number no longer measures any actual part of the chip. with 80 billion transistors on an 814 mm² die, the SXM version draws up to 700 W and pairs 80 GB of HBM3 memory at 3.35 TB/s with fourth-generation tensor corestensor core, n.: A small, specialized part of an AI chip that does one job — multiplying small blocks of numbers together — far faster than ordinary circuits can. that added FP8FP8, abbr.: An 8-bit number format that uses a quarter of the memory of the standard 32-bit format, letting new AI chips train and run models roughly twice as fast as with 16 bits. through a 'Transformer Engine' managing precision layer by layer — about 990 dense bfloat16bfloat16, n.: A 16-bit number format invented at Google for AI. It keeps the huge range of the standard 32-bit format but drops precision, which neural networks can usually tolerate. teraFLOPS and twice that in FP8. NVLinkNVLink, prop. n.: Nvidia's private high-speed connection between its AI chips, much faster than ordinary computer connections, letting dozens of chips share memory and act almost like one giant chip. 4 gives each GPU 900 GB/s. The H100 trained many frontier models of 2023–2024, and cluster sizes ('100,000 H100s') became shorthand for computecompute, n.: The raw computing power used to train and run AI, usually measured by how many specialized chips are used and for how long. With data and algorithms, it is a main driver of AI progress.. Variants include the H200 (141 GB of HBM3e) and the export-restricted H800 and H20 for China; successors are the Blackwell (2024) and Rubin (2026) generations.
Their cluster is about 50,000 H100-equivalents, counting the Blackwells at their FP8 throughput.
Origin
Announced by Jensen Huang at Nvidia's GTC conference on 22 March 2022. Demand after ChatGPT's release in November 2022 made it the emblematic chip of the generative-AI boom; analysts estimated selling prices of $25,000–$30,000 in early 2024, and US export rules shaped its China-specific variants.
Usage note
Often used as a unit: 'H100-equivalents' normalize mixed fleets to the H100's throughput in industry and policy estimates.
Related
- GPUGPU, abbr.: A computer chip first designed to draw video-game graphics that turned out to be ideal for AI, because it can do thousands of simple calculations at the same time.
- tensor coretensor core, n.: A small, specialized part of an AI chip that does one job — multiplying small blocks of numbers together — far faster than ordinary circuits can.
- FP8FP8, abbr.: An 8-bit number format that uses a quarter of the memory of the standard 32-bit format, letting new AI chips train and run models roughly twice as fast as with 16 bits.
- high-bandwidth memoryhigh-bandwidth memory, n.: Very fast memory made by stacking memory chips on top of each other and placing them right next to the processor, so AI chips can be fed data quickly enough.
- NVLinkNVLink, prop. n.: Nvidia's private high-speed connection between its AI chips, much faster than ordinary computer connections, letting dozens of chips share memory and act almost like one giant chip.
- export controls on AI chipsexport controls on AI chips, n.: Government rules that limit which countries and companies may buy the most advanced AI chips and the machines used to make them. The United States uses them mainly to slow China's progress in AI.
- GPU-hourGPU-hour, n.: One graphics chip working for one hour — the everyday unit for how much computing a job used and for renting AI chips by the hour.
- GPU poorGPU poor, adj.: Slang for researchers, startups or countries without access to the huge numbers of expensive chips needed to build the most advanced AI, as opposed to the GPU-rich big labs.
Sources
- NVIDIA (2022). NVIDIA H100 Tensor Core GPU Architecture. Whitepaper.
Mentioned in
GPUGPU, abbr.: A computer chip first designed to draw video-game graphics that turned out to be ideal for AI, because it can do thousands of simple calculations at the same time., Jensen HuangJensen Huang, prop. n.: The Taiwanese-American co-founder and chief executive of Nvidia, whose graphics chips became the main engine for training and running modern AI., NvidiaNvidia, prop. n.: An American chip company whose graphics processors became the main hardware for training and running AI, making it one of the most valuable companies in the world., AI supercomputerAI supercomputer, n.: A huge network of AI chips — often tens or hundreds of thousands — wired together in a data center to train the largest AI models., FlashAttentionFlashAttention, prop. n.: A faster way for AI chips to compute attention, the step in which a language model relates every word to every other. It gets exactly the same answer while moving far less data around the chip., FLOPSFLOPS, abbr.: Floating-point operations per second: how many arithmetic steps a computer can do each second. It is the standard speed rating for supercomputers and AI chips., FP8FP8, abbr.: An 8-bit number format that uses a quarter of the memory of the standard 32-bit format, letting new AI chips train and run models roughly twice as fast as with 16 bits., GPU-hourGPU-hour, n.: One graphics chip working for one hour — the everyday unit for how much computing a job used and for renting AI chips by the hour., high-bandwidth memoryhigh-bandwidth memory, n.: Very fast memory made by stacking memory chips on top of each other and placing them right next to the processor, so AI chips can be fed data quickly enough., memory bandwidthmemory bandwidth, n.: How fast data can be moved between a chip's memory and its calculators. For chatbots, this often decides how quickly words appear, more than raw calculating speed., arithmetic intensityarithmetic intensity, n.: How much calculating a task does for each piece of data it fetches from memory. Tasks with little calculation per fetch are slowed by memory, not by the chip's math speed., GPU poorGPU poor, adj.: Slang for researchers, startups or countries without access to the huge numbers of expensive chips needed to build the most advanced AI, as opposed to the GPU-rich big labs., NVLinkNVLink, prop. n.: Nvidia's private high-speed connection between its AI chips, much faster than ordinary computer connections, letting dozens of chips share memory and act almost like one giant chip., SRAMSRAM, abbr.: The fastest kind of computer memory, built directly into processor chips. It is quick but takes up a lot of space, so chips hold only a little of it.