AI stack

Engineering

The layered set of technologies used to build and run AI applications. A common breakdown runs from compute hardware (GPUsGPU, abbr.: A computer chip first designed to draw video-game graphics that turned out to be ideal for AI, because it can do thousands of simple calculations at the same time., TPUsTPU, abbr.: Google's own family of chips built specifically for AI, used to train and run its Gemini models and rented to other companies through its cloud. and other acceleratorsAI accelerator, n.: Any chip built or tuned to speed up AI — graphics chips, Google's TPUs, phone 'neural engines' and many startup designs. Such chips do the heavy math of training and running models.) and cloud infrastructure, through systems software such as drivers, CUDACUDA, prop. n.: Nvidia's software system for programming its graphics chips to do general calculations. Most AI software is built on it, which is a big reason Nvidia chips dominate AI. and compilers, and ML frameworks such as PyTorchPyTorch, prop. n.: The most widely used free software toolkit for building and training neural networks, especially in research; programmers write ordinary Python and it handles the calculus and the graphics chips. and JAXJAX, prop. n.: A free Google library that lets researchers write mathematics in familiar Python and automatically get derivatives and very fast code for graphics chips and Google's AI chips., to foundation modelsfoundation model, n.: A very large AI model, built once at great cost from vast amounts of text, images or code gathered mostly from the internet, that many products then reuse as their starting point, from chatbots to coding tools. used through APIsmodel API, n.: A web service through which programs send requests to an AI model run by a company and receive its answers, usually paying per use instead of running the model themselves. or self-hosted; then an application layer of orchestrationorchestration, n.: Coordinating several AI calls, tools or agents so each does its part in the right order and their results are combined into one finished piece of work, as a conductor directs an orchestra. frameworks, vector databasesvector database, n.: A database built to store embeddings — lists of numbers that represent meaning — and to quickly find the stored items most similar to a new one. It is the memory behind many AI search tools., retrievalretrieval-augmented generation, n.: Giving an AI an open-book exam: before answering, it looks up relevant passages in a set of documents and bases its reply on them, often citing where each point came from., gatewaysAI gateway, n.: A checkpoint between an organization's apps and the AI models they use that handles access, costs, logging, safety checks and switching between providers in one place., evaluation and observabilityLLM observability, n.: Recording and inspecting what an AI application actually did on each request — prompts sent, documents found, tools used, answers given, time and cost — so problems can be found and fixed. tools; and finally user-facing products. Engineers also use the phrase for a team's chosen components, and diagrams of the 'emerging LLM app stack' circulated widely from 2023.

Most of our AI stack is off the shelf; the only layer we build ourselves is evaluation.

Industry and policy

The value chain of the AI industry viewed as layers — chips, compute and cloud, data, models and applications — used to analyze market concentration, supply-chain dependence and national strategy, as in debates over which countries or companies control each layer and in policies that promote a 'full-stack' domestic AI industry or the export of a national AI technology stack.