Hardware & Compute

Chips, memory, interconnects, data centers and number formats. 71 entries live here, and 111 from other domains touch it.

Essential

AI acceleratorAny chip built or tuned to speed up AI — graphics chips, Google's TPUs, phone 'neural engines' and many startup designs. Such chips do the heavy math of training and running models.
AI data centerA building or campus packed with AI chips, power equipment and cooling, built to train and run AI models. The biggest use as much electricity as a city.
computeThe raw computing power used to train and run AI, usually measured by how many specialized chips are used and for how long. With data and algorithms, it is a main driver of AI progress.
FLOPA single arithmetic step, such as multiplying or adding two numbers. Counting them measures how much computing went into an AI model, and laws use such counts to decide which models face extra rules.
GPUA computer chip first designed to draw video-game graphics that turned out to be ideal for AI, because it can do thousands of simple calculations at the same time.
Moore's lawGordon Moore's prediction that chipmakers could pack twice as many transistors onto a chip roughly every two years — a pace that for decades made computers steadily faster and cheaper, and has slowed but not stopped.

All other entries

advanced packagingWays of joining several chips very closely inside one package — side by side on a silicon base or stacked on top of each other — so they act almost like one bigger chip.
AI compilerSoftware that translates an AI model into efficient instructions for a specific chip, rearranging and combining its calculations so the hardware is used as fully as possible.
AI factoryA data center built to turn electricity and data into AI models and AI answers at industrial scale; in Europe, also a public supercomputing hub that helps companies and researchers build AI.
AI supercomputerA huge network of AI chips — often tens or hundreds of thousands — wired together in a data center to train the largest AI models.
analog in-memory computingDoing AI math inside the memory itself, using electrical currents through stored values, instead of shuttling numbers back and forth to a separate processor. It could save lots of energy but is imprecise.
arithmetic intensityHow much calculating a task does for each piece of data it fetches from memory. Tasks with little calculation per fetch are slowed by memory, not by the chip's math speed.
ASICA chip whose circuits are laid out to do one kind of work extremely well instead of running any program. AI companies design their own to run their models more cheaply than with off-the-shelf chips.
bfloat16A 16-bit number format invented at Google for AI. It keeps the huge range of the standard 32-bit format but drops precision, which neural networks can usually tolerate.
chipletOne of several smaller chips that are made separately and then joined inside a single package to work as one processor, which is cheaper than making one giant chip and lets makers reuse parts.
CPUThe main general-purpose chip in a computer, good at running many different programs step by step. In AI systems it usually coordinates the work while specialized chips do the heavy math.
CUDANvidia's software system for programming its graphics chips to do general calculations. Most AI software is built on it, which is a big reason Nvidia chips dominate AI.
Dennard scalingThe rule that as transistors shrank they also used proportionally less power, so chips got faster without getting hotter. It stopped working in the mid-2000s, ending the era of ever-faster single processors.
DRAMThe main working memory of computers and AI chips: dense and fairly cheap, but slower than on-chip memory and needing constant refreshing to keep its data.
extreme ultraviolet lithographyThe chip-printing technique used for the most advanced chips, which draws circuit patterns with ultra-short-wavelength light. A single Dutch company, ASML, makes all the machines, each costing well over $100 million.
floating-point formatA way of storing numbers in a computer like scientific notation — a few digits plus a power of two — trading accuracy for range. AI uses ever-smaller formats to compute faster.
FLOPSFloating-point operations per second: how many arithmetic steps a computer can do each second. It is the standard speed rating for supercomputers and AI chips.
FP16A 16-bit way of storing numbers that uses half the memory of the standard 32-bit format. It is fast for AI but can't represent very large or very tiny values.
FP32The standard 32-bit way computers store numbers with fractional parts, keeping about seven significant digits. It was the default for training AI until faster, smaller formats took over.
FP4A tiny 4-bit number format that can represent only sixteen different values. With clever scaling tricks it lets the newest AI chips run, and even train, models faster and with less memory.
FP8An 8-bit number format that uses a quarter of the memory of the standard 32-bit format, letting new AI chips train and run models roughly twice as fast as with 16 bits.
FPGAA chip whose circuits can be rewired by software after it is made, so engineers can turn it into a custom processor — slower than a purpose-built chip, but flexible.
GPGPUUsing a graphics chip for ordinary number-crunching instead of drawing pictures — the trick that, years later, made modern AI training practical.
GPU-hourOne graphics chip working for one hour — the everyday unit for how much computing a job used and for renting AI chips by the hour.
H100Nvidia's flagship AI chip of 2022–2024, so scarce and sought-after during the generative-AI boom that it became a unit of measure for AI computing power.
hardware lotteryThe idea that research ideas succeed partly by luck — because they happen to suit the computer chips available at the time — rather than purely because they are better.
high-bandwidth memoryVery fast memory made by stacking memory chips on top of each other and placing them right next to the processor, so AI chips can be fed data quickly enough.
Huang's lawA claim, named after Nvidia's chief executive, that AI chip performance improves much faster than Moore's law because chips, software and number formats all improve together.
hyperscalerOne of the few giant technology companies — such as Amazon, Microsoft, Google and Meta — that run vast networks of data centers and spend most of the money that builds AI computing.
InfiniBandA very fast networking technology used to connect the thousands of servers in supercomputers and AI training clusters, designed so data moves with minimal delay.
INT8Storing numbers as whole numbers from −128 to 127 using 8 bits. AI models converted to this format run faster and need a quarter of the memory of 32-bit versions.
interconnectThe wiring and networking that connect chips to each other — inside a server, across a data center, or within a single chip — so they can share data and work as one.
liquid coolingCooling computer chips by running liquid past them instead of blowing air. AI chips now run so hot and are packed so tightly that air alone can no longer keep them cool.
memory bandwidthHow fast data can be moved between a chip's memory and its calculators. For chatbots, this often decides how quickly words appear, more than raw calculating speed.
memory wallThe growing gap between how fast chips can calculate and how fast they can fetch data from memory. Processors increasingly sit idle, waiting for numbers to arrive.
microscaling formatsNumber formats that store small groups of numbers with one shared scale factor, so each number can use very few bits while the group as a whole stays accurate.
MLPerfA standard set of speed tests for AI hardware and software, where chip makers and cloud companies compare how fast their systems train and run AI models.
model FLOPs utilizationThe share of a computer's top calculating speed that actually goes into useful AI training work. Real systems often reach only around a third to a half.
neocloudA newer kind of cloud company that mainly rents out powerful AI chips, rather than the full range of computing services that giants like Amazon or Microsoft sell.
neuromorphic computingComputer chips designed to work more like brains, with artificial neurons that send brief spikes only when needed, aiming to do AI tasks with far less energy.
NPUA small AI chip, or part of a chip, built into phones, laptops and other devices so they can run AI features quickly without draining the battery or using the cloud.
NVLinkNvidia's private high-speed connection between its AI chips, much faster than ordinary computer connections, letting dozens of chips share memory and act almost like one giant chip.
orbital data centerA proposed computer center circling the Earth, powered by almost constant sunlight, that would run AI without using land, water or power grids on the ground.
petaflop/s-dayA unit of computing work: what a computer doing a thousand trillion calculations per second gets done in one day. It is used to compare the cost of training AI models.
photonic computingUsing light instead of electricity to compute or move data. Light can carry huge amounts of information with little energy, but building full computers from it is still very hard.
power usage effectivenessA score showing how much of a data center's electricity actually reaches its computers: total power drawn divided by the computers' share. A perfect 1.0 would mean no waste; the extra goes mainly to cooling.
process nodeThe label for a generation of chip-making technology, like '3 nanometer'. Smaller numbers mean newer, denser chips, but the number no longer measures any actual part of the chip.
quantum machine learningResearch into whether quantum computers could learn from data faster or better than ordinary computers, and into using AI to study quantum systems. Practical advantages have not yet been shown.
rack-scale systemAn AI computer designed as one whole cabinet of chips, cooling and wiring, rather than separate servers, so all the chips inside can work together like one big chip.
RDMAA way for one computer to read or write another computer's memory directly over the network, skipping the usual software middlemen, so data moves faster with less delay.
ROCmAMD's free, open-source software for programming its graphics and AI chips — its answer to Nvidia's CUDA, letting AI software run on AMD hardware.
roofline modelA simple chart showing the fastest a program can run on a chip: limited either by how quickly data arrives from memory or by the chip's top calculating speed.
scale-up and scale-outTwo ways to get more computing power: make each machine bigger and more tightly linked (scale up), or add more machines connected by a network (scale out).
semiconductor foundryA factory company that manufactures chips designed by other companies. Taiwan's TSMC, the largest, makes nearly all of the world's most advanced AI chips.
silent data corruptionWhen computer hardware quietly produces wrong numbers without crashing or raising any alarm. In huge AI training runs on thousands of chips, such hidden errors can damage a model before anyone notices.
SRAMThe fastest kind of computer memory, built directly into processor chips. It is quick but takes up a lot of space, so chips hold only a little of it.
stragglerOne slow computer or chip in a large group working in lockstep. Because everyone must wait for the slowest member at each step, a single straggler can slow down an entire AI training job.
systolic arrayA grid of tiny calculators that pass numbers to their neighbors in rhythm, like a heartbeat, so each value is reused many times. AI chips use it to multiply large tables of numbers efficiently.
tensor coreA small, specialized part of an AI chip that does one job — multiplying small blocks of numbers together — far faster than ordinary circuits can.
TOPSTrillions of operations per second: the speed rating usually given for AI chips in phones and laptops, often counting simple whole-number math rather than precise fractions.
TPUGoogle's own family of chips built specifically for AI, used to train and run its Gemini models and rented to other companies through its cloud.
TritonA free language from OpenAI for writing the fast inner routines of AI programs in Python-style code; its compiler handles hardware details that Nvidia's own tools leave to experts.
von Neumann bottleneckThe traffic jam caused by ordinary computers keeping data and processing in separate places, so information must constantly shuttle back and forth through one narrow channel.
VRAMThe memory on a graphics card. For running AI at home it is the key number: the model has to fit in it, so more memory means bigger models.
wafer-scale engineA single giant processor made from nearly a whole dinner-plate-sized silicon wafer instead of the small chips normally cut from it, so data rarely has to leave the chip.
XLAA compiler from Google that turns AI programs into fast code for chips such as TPUs and GPUs, mainly by merging many small steps into fewer, bigger ones.

Also relevant

111

3D parallelismCombining three ways of dividing up training work — by data, by layer stage and by slices within layers — so that thousands of chips can train one giant model efficiently.
actuatorThe 'muscle' of a machine: a motor or similar device that turns electrical, hydraulic or air power into movement, such as turning a robot's joint or closing its gripper.
AI bubbleThe worry that investors are pouring far more money into AI companies, chips and data centers than AI will earn back soon, so that prices could crash as in the dot-com bust of 2000.
AI Diffusion RuleA 2025 US rule, cancelled before it took effect, that would have sorted the world's countries into three groups with different limits on how many advanced AI chips they could buy.
AI stackThe layers of technology an AI product depends on, from chips and data centers at the bottom, through the models, up to the apps people use, each built on the layer below.
all-reduceA group operation in which every chip contributes a set of numbers, the sets are added together, and every chip ends up with the same total. It is how parallel training keeps model copies in sync.
AMDAn American chip company that makes computer processors and graphics chips, and Nvidia's main rival in the chips used for artificial intelligence.
architectureThe overall design of an AI model, meaning what kinds of parts it has and how they are connected, as opposed to the specific numbers it learns during training.
ASMLA Dutch maker of the giant machines that print circuit patterns onto chips. It alone can build the extreme-ultraviolet models on which today's most advanced chips, including AI chips, depend.
batch sizeHow many training examples a model looks at before each adjustment of its settings. Bigger batches use hardware efficiently but bring diminishing returns; smaller ones are noisier but cheaper per step.
benchmarkA fixed, shared test — a set of questions or tasks with an agreed way of scoring — used to compare AI systems with one another and to track progress over time.
big dataData that piles up too fast, in too many forms or in too great a quantity for one ordinary computer to store and study, and the methods invented to cope with it.
biological anchorsA way of forecasting when AI could transform the world, by estimating how much computing it would take to match the human brain and when that much computing would be affordable.
BitNetA family of AI language models from Microsoft in which most of the learned numbers are limited to minus one, zero or one, so they need far less memory and energy to run.
CerebrasAn American chip company that makes a single giant processor the size of a whole silicon wafer, used to train and run AI models very quickly.
CERN for AIA proposed publicly funded, international AI laboratory modeled on CERN, the European particle-physics center, where countries would pool money, computers and scientists to do AI research together.
CHIPS and Science ActA 2022 US law that put tens of billions of dollars into building computer-chip factories and chip research in the United States, to reduce dependence on chips made in Asia.
circular dealA business arrangement in which an AI chip or cloud supplier invests in, or lends to, a customer that then spends the money buying the supplier's own products.
cognitive computingA label, promoted heavily by IBM in the 2010s, for computer systems that learn, reason and interact with people in humanlike ways. It was largely a marketing term and has given way to 'AI'.
collective communicationThe standard patterns by which a whole group of chips exchanges data at once — for example, everyone adding up their numbers and all getting the total — which large training jobs use constantly.
compute budgetThe total amount of computing a team can spend on training (or running) a model, set in advance like a money budget and used to decide how big the model should be and how much data it sees.
compute governanceUsing control over the computer chips and data centers that AI depends on — who can buy them, where they go, how they are used — as a way to oversee powerful AI.
compute overhangA situation in which far more computing power is available than AI currently uses, so that a new method or a decision to scale up could suddenly produce a big jump in ability.
confidential computingRunning computations inside a hardware-sealed compartment of a chip, so that data and AI models stay protected even from the cloud company or administrator operating the machine.
context parallelismA way of training or running a model on extremely long inputs — whole books or codebases — by splitting the text across many chips and having them swap the information each part needs to 'see' the others.
Core MLApple's software framework that lets apps on iPhones, iPads and Macs run AI models directly on the device, using its own chips instead of sending data to the cloud.
CoreWeaveAn American company that rents out huge numbers of Nvidia graphics processors over the internet to AI developers such as Microsoft, OpenAI and Meta.
critical batch sizeThe largest number of examples a model can usefully learn from in each training step. Below it, bigger batches let many chips work in parallel and finish sooner; above it, the extra chips are mostly wasted.
data parallelismSplitting the training data among many chips that each hold a full copy of the model; every chip learns from its own share, and they pool what they learned after each step so all copies stay identical.
Jeff DeanAn American computer scientist who built much of Google's computing infrastructure, co-founded the Google Brain deep-learning team and was Google's chief scientist before leaving in 2026 to co-found a startup.
decentralized trainingTraining an AI model on computers owned by many different people or companies around the world, linked over the ordinary internet rather than housed together in one data center.
decode phaseThe second stage of an AI's reply, in which it writes the answer one word-piece at a time, each step depending on everything written so far.
Deep BlueIBM's chess computer that beat world champion Garry Kasparov in 1997, the first time a machine defeated a reigning world champion in a match under standard tournament conditions.
DeepSpeedFree software from Microsoft for training and running AI models too big for ordinary tools, spread over many chips; it is best known for a memory-saving technique called ZeRO.
DiLoCoA training method from Google DeepMind that lets groups of chips in different places train one language model while talking to each other hundreds of times less often than usual.
disaggregated servingRunning the two stages of an AI's reply, reading the prompt and writing the answer, on separate groups of chips, each tuned for its own job, and handing the work between them.
distributed trainingTraining one AI model on many computer chips at once, splitting up the data or the model itself among them and keeping them in sync, so that jobs that would take a single chip thousands of years finish in weeks.
edge AIDoing AI's computing on or near the devices that collect the data, such as cameras, cars, factory machines, phones or local servers, instead of sending everything off to distant data centers.
environmental impact of AIThe effects of AI on the planet: the electricity, water, materials and land used to make chips, build data centers and train and run models, and the pollution that results.
ExecuTorchMeta's free software for running PyTorch AI models directly on phones, wearables and other small devices, without needing an internet connection or a data center.
expert parallelismA way of running a 'mixture of experts' model — one made of many specialist sub-networks — by giving each chip its own set of specialists and sending each piece of text to whichever chip holds the specialists it needs.
export controls on AI chipsGovernment rules that limit which countries and companies may buy the most advanced AI chips and the machines used to make them. The United States uses them mainly to slow China's progress in AI.
Fifth Generation Computer Systems projectJapan's government-backed effort, from 1982 to 1992, to build a new kind of computer that reasons with logic using many processors at once. It fell short, but it alarmed other countries into launching AI programs of their own.
FlashAttentionA faster way for AI chips to compute attention, the step in which a language model relates every word to every other. It gets exactly the same answer while moving far less data around the chip.
full fine-tuningFine-tuning in which every one of a model's internal numbers is allowed to change — the most flexible option, but also the most expensive in memory and storage.
fully sharded data parallelA feature of the popular PyTorch software that lets many chips train a model too big for any one of them, by spreading the model and its training bookkeeping evenly across all the chips.
goodputThe share of a big training job's time and computing that actually moves training forward, after subtracting time lost to crashes, restarts, waiting and repeated work. When serving users, it means the rate of requests answered fast enough.
GPU poorSlang for researchers, startups or countries without access to the huge numbers of expensive chips needed to build the most advanced AI, as opposed to the GPU-rich big labs.
gradient checkpointingA memory-saving trick: instead of storing every intermediate result needed for learning, the computer keeps only a few and recalculates the rest when needed, trading extra computing time for memory.
Jensen HuangThe Taiwanese-American co-founder and chief executive of Nvidia, whose graphics chips became the main engine for training and running modern AI.
HuaweiA Chinese technology giant, known for telecom equipment and phones, that designs China's leading homegrown AI chips despite US trade restrictions.
inference costWhat it costs to use an AI model after it has been built: the computing power, electricity and hardware needed every time it answers, usually billed per chunk of text.
JAXA free Google library that lets researchers write mathematics in familiar Python and automatically get derivatives and very fast code for graphics chips and Google's AI chips.
Jevons paradoxThe observation that making something more efficient can make people use more of it overall because it gets cheaper — as with coal in the 1800s, and with AI computing today.
kernelA word with several AI meanings: a similarity measure used by some learning methods; a small grid of numbers that scans an image inside a neural network; or a small program that runs on a GPU.
kernel fusionMerging several small calculation steps into one, so a chip finishes them in a single pass instead of saving and reloading data in between, like cooking at the counter without returning ingredients to the fridge after each step.
know-your-customer for computeRules that would make cloud companies check who their customers are before renting them enough computing power to train a large AI model, much as banks must verify who opens an account.
KV cacheA memory store where a chatbot keeps its working notes on everything already read and written, so each new word does not require re-reading the whole conversation from scratch.
latencyHow long you wait between asking an AI system for something and getting the answer. Lower latency means a snappier, more responsive system.
law of accelerating returnsRay Kurzweil's claim that technology improves at an ever-faster rate, because each advance helps build the next, so progress speeds up exponentially rather than steadily.
lidarA sensor that measures distances by firing laser pulses and timing their reflections, building a 3D picture of its surroundings out of millions of points; widely used on robots and self-driving cars.
Lisp machineA 1970s–1980s personal workstation built specially to run the AI programming language Lisp quickly. When ordinary computers caught up, the market collapsed, helping to bring on an AI winter.
LiteRTGoogle's free software for running AI models on phones, small gadgets and microcontrollers; it was called TensorFlow Lite until 2024 and is used inside many Android and iPhone apps.
Mark I PerceptronA room-sized machine built around 1960 that learned to recognize simple shapes and letters through a camera made of 400 light sensors. It was one of the first machines built to learn from examples.
matrixA rectangular grid of numbers arranged in rows and columns. Most of what a neural network has learned is stored in large grids like this.
matrix multiplicationThe rule for combining two grids of numbers into a new grid by repeatedly multiplying rows by columns. It is the single calculation AI chips spend most of their time doing.
Megatron-LMNVIDIA's open-source software, first described in a 2019 research paper, for training very large language models across thousands of graphics chips by splitting each model in several ways at once.
memory-boundDescribes a computer task slowed by how fast data can be fetched from memory rather than by how fast the chip can calculate — like a fast cook kept waiting for deliveries.
mixed-precision trainingTraining a model with shorter, less exact numbers for most of the arithmetic and full-length numbers only where accuracy really matters, which saves a great deal of memory and lets modern chips train far faster.
MLCommonsAn industry and university consortium that runs MLPerf, the standard tests of how fast computers train and run AI models, as well as benchmarks of AI safety.
MLXA free Apple software toolkit for running and training AI models efficiently on Mac computers with Apple's own chips, popular for running language models locally on a laptop.
model parallelismSplitting a single AI model that is too big for one chip into pieces spread over several chips, which must constantly pass intermediate results to each other as they work.
model weight securityProtecting the files that make up a trained AI model — its learned numbers — from being stolen or leaked, because whoever holds them can run, copy or alter the model.
National AI Research ResourceA US program to give university researchers, students and small companies shared access to the costly computers, data and AI models that are otherwise mostly available to big technology firms.
NCCLNVIDIA's software library that lets its graphics chips exchange data with each other quickly — inside one computer and across networks — and that nearly all large-scale AI training on NVIDIA hardware relies on.
N:M sparsityA tidy pattern of zeros inside an AI model, such as two zeros in every group of four numbers, that special chip hardware can skip to run the model faster.
numerical stabilityWhether a computer calculation stays accurate despite the tiny rounding errors in every step, instead of blowing up into nonsense such as infinities.
NvidiaAn American chip company whose graphics processors became the main hardware for training and running AI, making it one of the most valuable companies in the world.
offloadingMoving part of an AI model's data or work out of a chip's scarce fast memory onto ordinary memory, a disk or another computer, so that models too big for the chip can still run.
on-chip governance mechanismsFeatures built into AI chips that could check or enforce rules about how they are used — for example, proving which country a chip is in, or refusing to run without a valid license.
on-device AIAI that runs directly on your own phone, laptop, car or headset rather than on a company's distant servers, so it can work offline, respond instantly and keep your data on the device.
PagedAttentionA way of storing an AI model's working memory in small fixed-size blocks, like the pages of a computer's memory, so little space is wasted and more users can be served at once.
PaLMA very large Google language model from 2022 that showed sudden jumps in reasoning, and even explained jokes, as it grew. Its successor powered Google's Bard chatbot before the Gemini models replaced it.
parameter countHow many adjustable numbers an AI model contains — the usual measure of its size. Modern language models have billions to trillions of them.
parameter serverAn older design for training on many machines in which a few central computers hold the model and the others repeatedly fetch it, compute improvements on their share of data, and send them back.
pipeline parallelismSplitting a model's layers into consecutive stages on different chips, like stations on an assembly line, and streaming small batches of data through so that every station stays busy.
precisionOf all the items a model flagged as positive, the share that really were positive; a spam filter with high precision rarely sends genuine emails to the spam folder.
PyTorchThe most widely used free software toolkit for building and training neural networks, especially in research; programmers write ordinary Python and it handles the calculus and the graphics chips.
quantizationStoring an AI model's numbers with fewer digits, like rounding prices to the nearest dollar, so the model takes less memory and runs faster, usually with only a small loss of accuracy.
RayFree software that spreads Python programs, including AI training and model serving, across many computers and graphics chips so that big jobs run in parallel.
remote attestationA way for a computer to prove to someone far away exactly what software it is running, using a signature built into its hardware, so they can decide whether to trust it.
reservoir computingA shortcut way to build a network for time-varying signals: a large, fixed, randomly wired network echoes the input, and only a simple final readout is trained.
sequence parallelismSplitting the long string of words a model is working on into chunks handled by different chips, which saves memory and makes it possible to train on very long documents.
sovereign AIA country's ability to build and run AI on its own terms, so that the computers, models and know-how it relies on cannot be withheld or dictated by foreign companies or governments.
spiking neural networkA neural network that, like the brain, communicates with brief pulses, or spikes, sent only when a neuron's charge crosses a threshold. On special chips this can save a great deal of energy.
sponge exampleAn input crafted to make an AI system work as hard as possible — burning extra energy and time — to slow it down or run up costs, rather than to make it answer wrongly.
StargateA giant venture announced in 2025 by OpenAI, SoftBank and Oracle to build data centers in the United States for training and running AI, with a pledged $500 billion.
Strategic Computing InitiativeA billion-dollar US defense research program, run from 1983 to 1993, that tried to build machine intelligence for military uses such as self-driving vehicles and assistants for fighter pilots.
tensorA block of numbers with any number of dimensions: a single number, a list, a grid, or a stack of grids. AI software stores and processes all its data this way.
tensor parallelismSplitting the individual giant tables of numbers inside each layer of a model across several chips, so that every chip does a slice of every calculation at the same time.
TensorFlowGoogle's free software toolkit for building, training and running machine-learning models; it led the field after 2015 and still runs many models inside phones, browsers and company systems.
TensorRTNVIDIA's software for making a trained neural network run as fast as possible on NVIDIA graphics chips, by merging and simplifying its calculations and testing which way of doing each one is quickest.
TensorRT-LLMNVIDIA's free software library for running large language models as fast as possible on NVIDIA's own graphics chips, used by many companies that host AI models.
throughputHow much work an AI system gets done per second across all its users at once — for a chatbot service, roughly how much text it can produce in total each second.
TinyMLMachine learning shrunk to run on tiny, cheap, battery-powered chips inside sensors, wearables or appliances, using so little power that a coin-sized battery can last for months or years.
tokens per secondA speed rating for AI text generation: how many word-pieces a model produces each second. Five to ten keeps up with a fast reader; hundreds feel instant.
training computeThe total number of arithmetic operations performed to train a model. It is the standard yardstick for the size of an AI training effort, and some laws use it to decide which models face extra rules.
training runOne complete attempt at training a model with a fixed recipe, from start to finish. For the largest AI models a single run can occupy tens of thousands of chips for months.
trusted execution environmentA sealed-off part of a computer chip that runs programs which the rest of the machine, even its operating system, cannot spy on or interfere with.
TSMCThe Taiwanese company that manufactures most of the world's advanced computer chips, including nearly all of the specialized chips used to train and run AI.
ZeROA memory-saving technique for training big models on many chips: instead of every chip keeping its own complete copy of all the training bookkeeping, the chips divide it among themselves and share pieces only when needed.