NPU
In plain wordsA small AI chip, or part of a chip, built into phones, laptops and other devices so they can run AI features quickly without draining the battery or using the cloud.
An AI acceleratorAI accelerator, n.: Any chip built or tuned to speed up AI — graphics chips, Google's TPUs, phone 'neural engines' and many startup designs. Such chips do the heavy math of training and running models., usually a block within a system-on-chip, specialized for low-power neural-network inferenceinference, n.: Drawing a conclusion from evidence. In today's AI it most often means running an already-trained model to get an answer, as opposed to training it. on devices. Typical designs combine arrays of multiply–accumulate units for INT8INT8, abbr.: Storing numbers as whole numbers from −128 to 127 using 8 bits. AI models converted to this format run faster and need a quarter of the memory of 32-bit versions. and lower-precision arithmetic, local SRAMSRAM, abbr.: The fastest kind of computer memory, built directly into processor chips. It is quick but takes up a lot of space, so chips hold only a little of it. buffers that minimize memory traffic, and fixed-function support for common layers. NPUs are rated in TOPSTOPS, abbr.: Trillions of operations per second: the speed rating usually given for AI chips in phones and laptops, often counting simple whole-number math rather than precise fractions.; examples include Apple's Neural Engine, Qualcomm's Hexagon NPU, the NPUs in Intel and AMD laptop processors, and Google's Tensor phone chips. They make on-device AIon-device AI, n.: AI that runs directly on your own phone, laptop, car or headset rather than on a company's distant servers, so it can work offline, respond instantly and keep your data on the device. practical — photo processing, speech recognition, translation and small language models — while keeping data local and battery drain low. Some data-center chips (such as Huawei's Ascend) are also called NPUs, so the term names a market category more than a distinct architecture.
Origin
An early use for a dedicated neural-network accelerator is the 2012 MICRO paper by Hadi Esmaeilzadeh, Adrian Sampson, Luis Ceze and Doug Burger on neural acceleration of approximate programs. Dedicated NPUs reached consumers in 2017 in Huawei's Kirin 970 (marketed as having an 'NPU') and Apple's A11 (its 'Neural Engine'), and PCs in 2023–2024; Microsoft required at least 40 NPU TOPS for its Copilot+ PC label (May 2024).
Usage note
Vendor TOPS figures for NPUs typically assume INT8 (sometimes sparse) arithmetic and are not directly comparable with GPU FLOPS ratings.
Related
- AI acceleratorAI accelerator, n.: Any chip built or tuned to speed up AI — graphics chips, Google's TPUs, phone 'neural engines' and many startup designs. Such chips do the heavy math of training and running models.
- on-device AIon-device AI, n.: AI that runs directly on your own phone, laptop, car or headset rather than on a company's distant servers, so it can work offline, respond instantly and keep your data on the device.
- edge AIedge AI, n.: Doing AI's computing on or near the devices that collect the data, such as cameras, cars, factory machines, phones or local servers, instead of sending everything off to distant data centers.
- TOPSTOPS, abbr.: Trillions of operations per second: the speed rating usually given for AI chips in phones and laptops, often counting simple whole-number math rather than precise fractions.
- ASICASIC, abbr.: A chip whose circuits are laid out to do one kind of work extremely well instead of running any program. AI companies design their own to run their models more cheaply than with off-the-shelf chips.
- quantizationquantization, n.: Storing an AI model's numbers with fewer digits, like rounding prices to the nearest dollar, so the model takes less memory and runs faster, usually with only a small loss of accuracy.
- GPUGPU, abbr.: A computer chip first designed to draw video-game graphics that turned out to be ideal for AI, because it can do thousands of simple calculations at the same time.
Sources
- Esmaeilzadeh, H., Sampson, A., Ceze, L. & Burger, D. (2012). Neural Acceleration for General-Purpose Approximate Programs. Proceedings of MICRO-45.
Mentioned in
AI acceleratorAI accelerator, n.: Any chip built or tuned to speed up AI — graphics chips, Google's TPUs, phone 'neural engines' and many startup designs. Such chips do the heavy math of training and running models., on-device AIon-device AI, n.: AI that runs directly on your own phone, laptop, car or headset rather than on a company's distant servers, so it can work offline, respond instantly and keep your data on the device., ASICASIC, abbr.: A chip whose circuits are laid out to do one kind of work extremely well instead of running any program. AI companies design their own to run their models more cheaply than with off-the-shelf chips., CPUCPU, abbr.: The main general-purpose chip in a computer, good at running many different programs step by step. In AI systems it usually coordinates the work while specialized chips do the heavy math., edge AIedge AI, n.: Doing AI's computing on or near the devices that collect the data, such as cameras, cars, factory machines, phones or local servers, instead of sending everything off to distant data centers., INT8INT8, abbr.: Storing numbers as whole numbers from −128 to 127 using 8 bits. AI models converted to this format run faster and need a quarter of the memory of 32-bit versions., Core MLCore ML, prop. n.: Apple's software framework that lets apps on iPhones, iPads and Macs run AI models directly on the device, using its own chips instead of sending data to the cloud., ExecuTorchExecuTorch, prop. n.: Meta's free software for running PyTorch AI models directly on phones, wearables and other small devices, without needing an internet connection or a data center., LiteRTLiteRT, prop. n.: Google's free software for running AI models on phones, small gadgets and microcontrollers; it was called TensorFlow Lite until 2024 and is used inside many Android and iPhone apps., ONNXONNX, prop. n.: A shared file format for trained AI models, so that a model built with one toolkit can be moved to and run by another, much as a PDF opens on any computer., systolic arraysystolic array, n.: A grid of tiny calculators that pass numbers to their neighbors in rhythm, like a heartbeat, so each value is reused many times. AI chips use it to multiply large tables of numbers efficiently., TOPSTOPS, abbr.: Trillions of operations per second: the speed rating usually given for AI chips in phones and laptops, often counting simple whole-number math rather than precise fractions.