AI accelerator
In plain wordsAny chip built or tuned to speed up AI — graphics chips, Google's TPUs, phone 'neural engines' and many startup designs. Such chips do the heavy math of training and running models.
A processor specialized for the arithmetic of neural networksneural network, n.: A kind of computer program that learns a task from examples by tuning thousands to billions of connections between simple number-crunching units, an arrangement loosely inspired by nerve cells in the brain. — chiefly dense matrix multiplicationmatrix multiplication, n.: The rule for combining two grids of numbers into a new grid by repeatedly multiplying rows by columns. It is the single calculation AI chips spend most of their time doing. at low precision — as opposed to a general-purpose CPUCPU, abbr.: The main general-purpose chip in a computer, good at running many different programs step by step. In AI systems it usually coordinates the work while specialized chips do the heavy math.. Accelerators trade flexibility for throughput and energy efficiency through massively parallel multiply–accumulate arrays (tensor corestensor core, n.: A small, specialized part of an AI chip that does one job — multiplying small blocks of numbers together — far faster than ordinary circuits can., systolic arrayssystolic array, n.: A grid of tiny calculators that pass numbers to their neighbors in rhythm, like a heartbeat, so each value is reused many times. AI chips use it to multiply large tables of numbers efficiently.), reduced-precision formats (bfloat16bfloat16, n.: A 16-bit number format invented at Google for AI. It keeps the huge range of the standard 32-bit format but drops precision, which neural networks can usually tolerate., FP8FP8, abbr.: An 8-bit number format that uses a quarter of the memory of the standard 32-bit format, letting new AI chips train and run models roughly twice as fast as with 16 bits., INT8INT8, abbr.: Storing numbers as whole numbers from −128 to 127 using 8 bits. AI models converted to this format run faster and need a quarter of the memory of 32-bit versions.), large on-chip SRAMSRAM, abbr.: The fastest kind of computer memory, built directly into processor chips. It is quick but takes up a lot of space, so chips hold only a little of it. or stacked high-bandwidth memoryhigh-bandwidth memory, n.: Very fast memory made by stacking memory chips on top of each other and placing them right next to the processor, so AI chips can be fed data quickly enough., and fast chip-to-chip interconnectsinterconnect, n.: The wiring and networking that connect chips to each other — inside a server, across a data center, or within a single chip — so they can share data and work as one.. The category spans data-center GPUsGPU, abbr.: A computer chip first designed to draw video-game graphics that turned out to be ideal for AI, because it can do thousands of simple calculations at the same time. (Nvidia, AMD), custom ASICsASIC, abbr.: A chip whose circuits are laid out to do one kind of work extremely well instead of running any program. AI companies design their own to run their models more cheaply than with off-the-shelf chips. (Google's TPUTPU, abbr.: Google's own family of chips built specifically for AI, used to train and run its Gemini models and rented to other companies through its cloud., AWS Trainium, Huawei Ascend), wafer-scalewafer-scale engine, n.: A single giant processor made from nearly a whole dinner-plate-sized silicon wafer instead of the small chips normally cut from it, so data rarely has to leave the chip. and SRAM-centric designs (Cerebras, Groq), and low-power NPUsNPU, abbr.: A small AI chip, or part of a chip, built into phones, laptops and other devices so they can run AI features quickly without draining the battery or using the cloud. in phones and laptops. Accelerators now perform nearly all AI training and most large-scale inference, which is why their supply is a focus of industrial policy and export controls on AI chipsexport controls on AI chips, n.: Government rules that limit which countries and companies may buy the most advanced AI chips and the machines used to make them. The United States uses them mainly to slow China's progress in AI..
The startup's accelerator beats GPUs on latency but only runs models that fit in its on-chip memory.
Origin
Dedicated neural-network chips date to the late 1980s (Intel's analog ETANN of 1989, for example) but were overtaken by general-purpose processors. The current wave began when GPUs proved effective for deep learning (2009–2012) and Google deployed its TPU (2015); a surge of startup and hyperscaler designs followed from 2016.
Usage note
'Accelerator' implies a host CPU that feeds the chip work. Vendor peak figures differ in precision and sparsity assumptions, so compare like with like.
Related
- GPUGPU, abbr.: A computer chip first designed to draw video-game graphics that turned out to be ideal for AI, because it can do thousands of simple calculations at the same time.
- TPUTPU, abbr.: Google's own family of chips built specifically for AI, used to train and run its Gemini models and rented to other companies through its cloud.
- NPUNPU, abbr.: A small AI chip, or part of a chip, built into phones, laptops and other devices so they can run AI features quickly without draining the battery or using the cloud.
- ASICASIC, abbr.: A chip whose circuits are laid out to do one kind of work extremely well instead of running any program. AI companies design their own to run their models more cheaply than with off-the-shelf chips.
- FPGAFPGA, abbr.: A chip whose circuits can be rewired by software after it is made, so engineers can turn it into a custom processor — slower than a purpose-built chip, but flexible.
- wafer-scale enginewafer-scale engine, n.: A single giant processor made from nearly a whole dinner-plate-sized silicon wafer instead of the small chips normally cut from it, so data rarely has to leave the chip.
- neuromorphic computingneuromorphic computing, n.: Computer chips designed to work more like brains, with artificial neurons that send brief spikes only when needed, aiming to do AI tasks with far less energy.
- computecompute, n.: The raw computing power used to train and run AI, usually measured by how many specialized chips are used and for how long. With data and algorithms, it is a main driver of AI progress.
- CPUCPU, abbr.: The main general-purpose chip in a computer, good at running many different programs step by step. In AI systems it usually coordinates the work while specialized chips do the heavy math.
Mentioned in
computecompute, n.: The raw computing power used to train and run AI, usually measured by how many specialized chips are used and for how long. With data and algorithms, it is a main driver of AI progress., export controls on AI chipsexport controls on AI chips, n.: Government rules that limit which countries and companies may buy the most advanced AI chips and the machines used to make them. The United States uses them mainly to slow China's progress in AI., GPUGPU, abbr.: A computer chip first designed to draw video-game graphics that turned out to be ideal for AI, because it can do thousands of simple calculations at the same time., Moore's lawMoore's law, n.: Gordon Moore's prediction that chipmakers could pack twice as many transistors onto a chip roughly every two years — a pace that for decades made computers steadily faster and cheaper, and has slowed but not stopped., NvidiaNvidia, prop. n.: An American chip company whose graphics processors became the main hardware for training and running AI, making it one of the most valuable companies in the world., AI stackAI stack, n.: The layers of technology an AI product depends on, from chips and data centers at the bottom, through the models, up to the apps people use, each built on the layer below., AI supercomputerAI supercomputer, n.: A huge network of AI chips — often tens or hundreds of thousands — wired together in a data center to train the largest AI models., AMDAMD, prop. n.: An American chip company that makes computer processors and graphics chips, and Nvidia's main rival in the chips used for artificial intelligence., ASICASIC, abbr.: A chip whose circuits are laid out to do one kind of work extremely well instead of running any program. AI companies design their own to run their models more cheaply than with off-the-shelf chips., CHIPS and Science ActCHIPS and Science Act, prop. n.: A 2022 US law that put tens of billions of dollars into building computer-chip factories and chip research in the United States, to reduce dependence on chips made in Asia., CPUCPU, abbr.: The main general-purpose chip in a computer, good at running many different programs step by step. In AI systems it usually coordinates the work while specialized chips do the heavy math., edge AIedge AI, n.: Doing AI's computing on or near the devices that collect the data, such as cameras, cars, factory machines, phones or local servers, instead of sending everything off to distant data centers., HuaweiHuawei, prop. n.: A Chinese technology giant, known for telecom equipment and phones, that designs China's leading homegrown AI chips despite US trade restrictions., model servingmodel serving, n.: Running a trained AI model as a live service, so that apps and people can send it requests and get answers back quickly, reliably and at a sensible cost., NPUNPU, abbr.: A small AI chip, or part of a chip, built into phones, laptops and other devices so they can run AI features quickly without draining the battery or using the cloud., TPUTPU, abbr.: Google's own family of chips built specifically for AI, used to train and run its Gemini models and rented to other companies through its cloud., AI compilerAI compiler, n.: Software that translates an AI model into efficient instructions for a specific chip, rearranging and combining its calculations so the hardware is used as fully as possible., analog in-memory computinganalog in-memory computing, n.: Doing AI math inside the memory itself, using electrical currents through stored values, instead of shuttling numbers back and forth to a separate processor. It could save lots of energy but is imprecise., CerebrasCerebras, prop. n.: An American chip company that makes a single giant processor the size of a whole silicon wafer, used to train and run AI models very quickly., Dennard scalingDennard scaling, n.: The rule that as transistors shrank they also used proportionally less power, so chips got faster without getting hotter. It stopped working in the mid-2000s, ending the era of ever-faster single processors., FPGAFPGA, abbr.: A chip whose circuits can be rewired by software after it is made, so engineers can turn it into a custom processor — slower than a purpose-built chip, but flexible., GPU poorGPU poor, adj.: Slang for researchers, startups or countries without access to the huge numbers of expensive chips needed to build the most advanced AI, as opposed to the GPU-rich big labs., hardware lotteryhardware lottery, n.: The idea that research ideas succeed partly by luck — because they happen to suit the computer chips available at the time — rather than purely because they are better., Huang's lawHuang's law, n.: A claim, named after Nvidia's chief executive, that AI chip performance improves much faster than Moore's law because chips, software and number formats all improve together.