compound AI system
In plain wordsAn AI product built from several working parts — models, search tools, code and checks — designed to work together, rather than a single model answering on its own.
An AI application whose answers come from several cooperating parts, such as retrievers, code interpreters and other toolstool, n.: Any outside capability an AI model is allowed to use — a web search, a calculator, a code runner, a database or an app — described to the model so it can ask for it by name., verifiers, control logic and calls to one or more models, rather than from a single call to one model. Retrieval-augmented generationretrieval-augmented generation, n.: Giving an AI an open-book exam: before answering, it looks up relevant passages in a set of documents and bases its reply on them, often citing where each point came from. pipelines, multi-step agentic workflowsagentic workflow, n.: A multi-step process in which AI models draft, check, use tools and revise their work over several rounds, instead of answering in one shot; often the steps are laid out by developers in advance., systems that sample many candidate solutions and filter them with tests, and routersmodel routing, n.: Automatically choosing which AI model should answer each request, a cheap quick one for easy questions and a bigger slower one for hard ones, to save money without hurting quality. that send queries to models of different sizes are all compound systems. Matei Zaharia and colleagues, who popularized the term, argue that the best results increasingly come from system design rather than model scale alone, because systems can add fresh or private information, enforce control and safety, trade cost against quality, and improve by tuning components together. It also brings engineering problems: a much larger design space, optimization across components that are not differentiable (addressed by tools such as DSPyDSPy, prop. n.: A free software framework in which programmers describe what each step of an AI pipeline should do, and the framework automatically works out good prompts and examples instead of hand-writing them.), and harder evaluation, monitoring and debugging.
Our support bot is a compound AI system: a router, two retrievers, a drafting model and a policy checker.
Origin
Popularized by Matei Zaharia, Omar Khattab and colleagues at the Berkeley Artificial Intelligence Research (BAIR) lab in the February 2024 blog post 'The Shift from Models to Compound AI Systems', which argued that leading AI results were increasingly obtained by systems rather than single models.
Usage note
Overlaps with agentic systemsagentic AI, n.: AI built to pursue goals on its own — planning, taking actions and adjusting along the way — instead of only responding to one prompt at a time. 'Agentic' describes how much of this independence a system has., but a compound system need not be agentic: its control flow can be fixed by developers. The legal term AI systemAI system, n.: A complete piece of technology that uses AI to do something, such as screening loan applications, driving a car or chatting with customers, including the AI model and the software around it. is broader still.
Related
- DSPyDSPy, prop. n.: A free software framework in which programmers describe what each step of an AI pipeline should do, and the framework automatically works out good prompts and examples instead of hand-writing them.
- retrieval-augmented generationretrieval-augmented generation, n.: Giving an AI an open-book exam: before answering, it looks up relevant passages in a set of documents and bases its reply on them, often citing where each point came from.
- agentic workflowagentic workflow, n.: A multi-step process in which AI models draft, check, use tools and revise their work over several rounds, instead of answering in one shot; often the steps are laid out by developers in advance.
- AI stackAI stack, n.: The layers of technology an AI product depends on, from chips and data centers at the bottom, through the models, up to the apps people use, each built on the layer below.
- orchestrationorchestration, n.: Coordinating several AI calls, tools or agents so each does its part in the right order and their results are combined into one finished piece of work, as a conductor directs an orchestra.
- model routingmodel routing, n.: Automatically choosing which AI model should answer each request, a cheap quick one for easy questions and a bigger slower one for hard ones, to save money without hurting quality.
- AI systemAI system, n.: A complete piece of technology that uses AI to do something, such as screening loan applications, driving a car or chatting with customers, including the AI model and the software around it.
Sources
- Zaharia, M., Khattab, O., Chen, L., Davis, J. Q., Miller, H., Potts, C., Zou, J., Carbin, M., Frankle, J., Rao, N. & Ghodsi, A. (2024). The Shift from Models to Compound AI Systems. BAIR blog, 18 February 2024. bair.berkeley.edu/blog/2024/02/18/compound-ai-systems
Mentioned in
AI engineeringAI engineering, n.: The craft of building useful, reliable products on top of AI models — choosing models, writing prompts, connecting data and tools, testing quality and running it all safely at scale., AI stackAI stack, n.: The layers of technology an AI product depends on, from chips and data centers at the bottom, through the models, up to the apps people use, each built on the layer below., DSPyDSPy, prop. n.: A free software framework in which programmers describe what each step of an AI pipeline should do, and the framework automatically works out good prompts and examples instead of hand-writing them.