Core ConceptsLanguage Models & TransformersTraining & Post-trainingGovernance, Policy & Law

foundation model

nounAlso: FM, large pretrained model, pretrained model

In plain wordsA very large AI model, built once at great cost from vast amounts of text, images or code gathered mostly from the internet, that many products then reuse as their starting point, from chatbots to coding tools.

A large model pretrained on an exceptionally wide range of data, typically by self-supervised learningself-supervised learning, n.: A way for AI to learn from huge amounts of unlabeled data by setting itself fill-in-the-blank puzzles, such as guessing a hidden word or a missing patch of an image. over enormous text, image or code corpora, so that one trained model can serve as the common base for many applications, each adapting it by fine-tuningfine-tuning, n.: Taking an AI model that has already learned a lot and training it a little more on a smaller, focused set of examples so it gets better at one job or behaves a certain way., promptingprompt, n.: The message or instructions you give an AI system — a question, a request, some examples or a document — that it responds to. Clearer prompts generally get better results. or in-context learningin-context learning, n.: A language model's ability to pick up a new task from instructions or examples placed in its input, on the spot, without any retraining — like a quick study learning a game's rules from one round.. What sets the class apart is this breadth of reuse, not a particular architecture. Examples include large language modelslarge language model, n.: A computer program trained on enormous amounts of text to predict the words that come next. At vast scale, that simple skill lets it chat, answer questions, translate, summarize and write code. such as GPT-3GPT-3, prop. n.: A huge 2020 OpenAI language model that could perform new tasks simply from instructions or a few examples written into its prompt, without retraining. It made prompting a new way of programming. and its successors, vision-language modelsvision-language model, n.: An AI model that understands images and text together — it can describe a photo, answer questions about a chart or find the picture that matches a sentence., CLIPCLIP, prop. n.: An OpenAI model that learned to match pictures with their captions from hundreds of millions of web examples, so it can judge how well any piece of text describes any image., and image and video generators. The coiners stressed two properties: emergence (capabilities arising implicitly from scale rather than being designed in) and homogenizationhomogenization, n.: The tendency of AI to make things more alike: products built on the same few models share the same flaws, and people who write or create with AI produce more similar work. (many applications built on the same few models, so their strengths and defects propagate downstream). Because foundation models are costly to train and come from a small number of developers, they are a focus of AI governanceAI governance, n.: The rules, institutions and habits that shape how AI is built and used — from laws and treaties to technical standards and a company's own review processes.; the EU AI Act regulates them under the name general-purpose AI modelgeneral-purpose AI model, n.: The EU's legal name for the large, adaptable AI models behind chatbots and similar tools: models that can turn their hand to many different jobs and be built into many other products..
Rather than train our own, we fine-tune an open foundation model on our support tickets.

Origin

Introduced by Rishi Bommasani, Percy Liang and more than 100 Stanford co-authors in 'On the Opportunities and Risks of Foundation Models' (August 2021), alongside the launch of Stanford's Center for Research on Foundation Models. They chose the word to underline the models' 'critically central yet incomplete character'. Critics objected that 'foundation' implied a soundness the models lacked; 'large pretrained model' is a neutral alternative.

Usage note

Overlaps with frontier modelfrontier model, n.: One of the most capable AI models in existence at a given time, built by a handful of leading labs at great cost, and the main target of AI safety rules. (the most capable foundation models at a given time), base modelbase model, n.: A language model fresh from its first broad training on text, before being taught to follow instructions or chat. It continues whatever text it is given instead of answering. (a model before post-training) and general-purpose AI modelgeneral-purpose AI model, n.: The EU's legal name for the large, adaptable AI models behind chatbots and similar tools: models that can turn their hand to many different jobs and be built into many other products. (the EU legal term). A large model trained for one narrow task is not a foundation model.

Sources

Bommasani, R. et al. (2021). On the Opportunities and Risks of Foundation Models. arXiv:2108.07258. arxiv.org/abs/2108.07258

Defined in law or standards

EU AI Act (Reg. (EU) 2024/1689), Art. 3(63) ('general-purpose AI model', the closest EU counterpart); US Executive Order 14110 (2023; revoked January 2025), §3(k) ('dual-use foundation model')

Mentioned in

computer visioncomputer vision, n.: The part of AI that lets computers make sense of images and video: recognizing faces, reading signs, spotting tumors on scans or judging how far away a car is., frontier AIfrontier AI, n.: The most capable general-purpose AI systems of the moment — those at or beyond the leading edge — which governments single out for extra safety attention., frontier modelfrontier model, n.: One of the most capable AI models in existence at a given time, built by a handful of leading labs at great cost, and the main target of AI safety rules., general-purpose AI modelgeneral-purpose AI model, n.: The EU's legal name for the large, adaptable AI models behind chatbots and similar tools: models that can turn their hand to many different jobs and be built into many other products., generative AIgenerative AI, n.: AI that produces something new on request, such as an essay, a picture, a song or a piece of code, having absorbed from enormous numbers of existing examples what such things typically look like., large language modellarge language model, n.: A computer program trained on enormous amounts of text to predict the words that come next. At vast scale, that simple skill lets it chat, answer questions, translate, summarize and write code., modelmodel, n.: A simplified working version of something. In AI it usually means the trained program, a big set of learned numbers, that takes an input and produces an answer, prediction or piece of content., pretrainingpretraining, n.: The first and usually the most expensive stage of building a modern AI model, in which it learns general patterns from enormous amounts of raw text, images or code before being specialized for particular uses., transfer learningtransfer learning, n.: Reusing what a model learned on one task to get a head start on another — like a pianist picking up the organ faster than a beginner. It lets models learn new jobs from little data., AI engineerAI engineer, n.: A software developer who builds products using ready-made AI models — connecting them to data and tools and testing that they work well — rather than inventing or training the models., AI engineeringAI engineering, n.: The craft of building useful, reliable products on top of AI models — choosing models, writing prompts, connecting data and tools, testing quality and running it all safely at scale., AI modelAI model, n.: The trained core of an AI: a large file of learned numbers plus the recipe for using them, which turns an input such as a question or a photo into an output., AI stackAI stack, n.: The layers of technology an AI product depends on, from chips and data centers at the bottom, through the models, up to the apps people use, each built on the layer below., AutoMLAutoML, n.: Software that automates the trial-and-error parts of building a machine-learning model, such as choosing the algorithm, its settings and the data-preparation steps, so less expert effort is needed., backbonebackbone, n.: The main, usually pretrained part of a vision model that turns an image into useful internal features; smaller task-specific parts are attached to it, like limbs to a spine., base modelbase model, n.: A language model fresh from its first broad training on text, before being taught to follow instructions or chat. It continues whatever text it is given instead of answering., G7 Hiroshima AI ProcessG7 Hiroshima AI Process, prop. n.: An effort begun by the G7 group of wealthy democracies in Japan in 2023 that sets voluntary rules for companies building the most advanced AI, which now report publicly on how they follow them., general-purpose AIgeneral-purpose AI, n.: AI that can do a wide range of different tasks, like writing, coding, translating and analyzing images, rather than the one job it was built for. Today's chatbots are the main example., homogenizationhomogenization, n.: The tendency of AI to make things more alike: products built on the same few models share the same flaws, and people who write or create with AI produce more similar work., narrow AInarrow AI, n.: AI built to do one specific job, or a small set of related jobs, such as playing chess or spotting spam, without being able to turn its skill to unrelated problems., perceptionperception, n.: How an AI system or robot turns raw sensor signals, such as camera pixels, microphone sound or laser scans, into an understanding of what is around it: objects, people, words and distances., pretraining datapretraining data, n.: The enormous body of text, code and other material — mostly from the web — that a large AI model first learns from, before it is tuned to follow instructions or chat., RAISE ActRAISE Act, prop. n.: A New York law requiring the largest AI developers to publish how they manage catastrophic risks and to report serious safety incidents quickly to a new state office. It takes effect in 2027., robot foundation modelrobot foundation model, n.: One large AI model trained on huge amounts of robot and internet data so that it can control many robots on many tasks, and be adapted quickly to new ones.