retrieval-augmented generation
In plain wordsGiving an AI an open-book exam: before answering, it looks up relevant passages in a set of documents and bases its reply on them, often citing where each point came from.
- 1
General
A system design in which, before a generative model answers, a retrieval step looks up material relevant to the request — passages from a document collection, database records, web-search results or API responses — and places it in the model's input, so that the answer can rest on that material and not only on what the model absorbed in training (parametric knowledgeparametric knowledge, n.: What an AI model 'knows' from training, stored in the numbers of its network rather than looked up in documents — like facts you have memorized rather than ones you check in a book.). A typical pipeline splits documents into passages (chunkingchunking, n.: Cutting long documents into smaller pieces before storing them for AI search, so the system can find and hand the model just the relevant part instead of a whole manual.), indexes them with text embeddingstext embedding, n.: A list of numbers that captures what a piece of text means, so that texts with similar meanings get similar lists. Computers use them to search, sort and group text by meaning. in a vector databasevector database, n.: A database built to store embeddings — lists of numbers that represent meaning — and to quickly find the stored items most similar to a new one. It is the memory behind many AI search tools. and/or a keyword index, retrieves candidates for each query by densedense retrieval, n.: Finding relevant documents by turning the question and every document into lists of numbers that capture meaning, then picking the documents whose numbers are closest to the question's., sparsesparse retrieval, n.: Finding documents by the words they contain, giving more weight to rare, telling words — the classic way search engines work. or hybrid searchhybrid search, n.: Searching two ways at once — by exact keywords and by meaning — and merging the results, so you catch precise matches like product codes as well as differently worded matches., optionally reranksreranking, n.: A second, more careful pass over search results: a fast method grabs a few hundred candidates, then a slower, smarter model reorders them so the best come first. them, and inserts the best into the prompt with instructions to answer from them and give citationscitation, n.: A pointer from something an AI says to the source that supports it — a link, document or quoted passage — so people can check the claim for themselves.. Because the knowledge sits in an index rather than in the weights, a RAG system can draw on private, fast-changing or very large collections without retraining and can show its sources so that answers can be checked; grounding also reduces, but does not eliminate, hallucinationhallucination, n.: When an AI states something false or made up — a fake fact, quotation or source — fluently and confidently, as if it were true.. Weak retrieval, badly chosen chunks and positional effectslost in the middle, phr.: AI models reading a long input behave a little like a skimmer: they reliably catch facts placed near its start or end but are much likelier to miss the same fact buried halfway through. are common failure points; in agentic RAGagentic RAG, n.: Retrieval in which the AI runs its own research: it decides what to search for, looks at the results, searches again if needed, and only then answers. the model steers retrieval itself.
We answer HR questions with RAG over the policy handbook, so policy updates take effect without retraining.
- 2
Original model
The model that introduced the name (Lewis et al., 2020). Its retriever encoded Wikipedia passages as dense vectors and fetched those closest to the encoded query; its generator, the pretrained sequence-to-sequencesequence-to-sequence model, n.: A model that turns one sequence into another, such as a French sentence into English or speech into text, by first reading the whole input and then writing the output step by step. model BART, conditioned on what was fetched. The retrieved passage was treated as a latent variable to be marginalized over, and training updated the query encoder and generator jointly while the passage index stayed fixed. Two variants differed in how often they consulted the passages: RAG-Sequence relied on one set of retrieved passages for an entire answer, whereas RAG-Token could switch passages at every token.
Origin
Named by Patrick Lewis and colleagues at Facebook AI Research, University College London and New York University in 'Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks' (arXiv, May 2020; NeurIPS 2020). Retrieve-then-read question answering such as DrQA (2017) and the retrieval-augmented language model REALM (2020) came before. After ChatGPT's release in late 2022, 'RAG' became the generic name for any LLM application that retrieves before it answers.
Usage note
Long context windowscontext window, n.: How much an AI model can read and keep in view at once: the instructions, the conversation so far and any attached documents. Text that does not fit has to be cut, summarized or left out. have not made RAG obsolete: retrieval stays cheaper, more current and more auditable for large or changing corpora, though for small collections loading everything into context can be simpler. The retrieval need not be vector search; keyword search, SQL and web search all qualify.
Related
- information retrievalinformation retrieval, n.: The study and engineering of search: how to organize huge piles of documents so that the few that matter to someone's question can be found and ranked quickly. AI systems rely on it to look things up.
- groundinggrounding, n.: Connecting an AI's words to something real — trusted documents, a live search, images or the physical world — so that its answers rest on evidence rather than only on what it absorbed in training.
- vector databasevector database, n.: A database built to store embeddings — lists of numbers that represent meaning — and to quickly find the stored items most similar to a new one. It is the memory behind many AI search tools.
- chunkingchunking, n.: Cutting long documents into smaller pieces before storing them for AI search, so the system can find and hand the model just the relevant part instead of a whole manual.
- rerankingreranking, n.: A second, more careful pass over search results: a fast method grabs a few hundred candidates, then a slower, smarter model reorders them so the best come first.
- agentic RAGagentic RAG, n.: Retrieval in which the AI runs its own research: it decides what to search for, looks at the results, searches again if needed, and only then answers.
- GraphRAGGraphRAG, prop. n.: A way of letting AI answer questions about a big pile of documents by first turning them into a map of people, things and their connections, then using that map to find and summarize answers.
- context engineeringcontext engineering, n.: Managing an AI model's reading material: choosing, trimming and arranging the instructions, documents, tool results and memories it is given on every turn, so that it has what the task needs and little else.
- fine-tuningfine-tuning, n.: Taking an AI model that has already learned a lot and training it a little more on a smaller, focused set of examples so it gets better at one job or behaves a certain way.
- parametric knowledgeparametric knowledge, n.: What an AI model 'knows' from training, stored in the numbers of its network rather than looked up in documents — like facts you have memorized rather than ones you check in a book.
Sources
- Lewis, P., Perez, E., Piktus, A., Petroni, F., Karpukhin, V., Goyal, N., Küttler, H., Lewis, M., Yih, W., Rocktäschel, T., Riedel, S. & Kiela, D. (2020). Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks. NeurIPS 2020. arXiv:2005.11401. arxiv.org/abs/2005.11401v4
Mentioned in
agent memoryagent memory, n.: The ways an AI assistant or agent keeps information — your preferences, past conversations, lessons from earlier tasks — and brings it back later, beyond what fits in a single conversation., context windowcontext window, n.: How much an AI model can read and keep in view at once: the instructions, the conversation so far and any attached documents. Text that does not fit has to be cut, summarized or left out., deep researchdeep research, n.: An AI feature that spends minutes rather than seconds on a question — planning, running dozens of web searches, reading many sources — and then writes a long report with citations., embeddingembedding, n.: A list of numbers that stands for a word, picture or other item, arranged so that similar items get similar lists. It lets a computer measure similarity of meaning as distance., hallucinationhallucination, n.: When an AI states something false or made up — a fake fact, quotation or source — fluently and confidently, as if it were true., knowledge cutoffknowledge cutoff, n.: The date after which an AI model knows nothing from its training, because its training data stops there. Without a web search tool, it cannot know about later events., trainingtraining, n.: The process by which an AI model learns: it is shown huge numbers of examples and its internal settings are adjusted, bit by bit, until its answers get better., agentic RAGagentic RAG, n.: Retrieval in which the AI runs its own research: it decides what to search for, looks at the results, searches again if needed, and only then answers., AI engineerAI engineer, n.: A software developer who builds products using ready-made AI models — connecting them to data and tools and testing that they work well — rather than inventing or training the models., AI engineeringAI engineering, n.: The craft of building useful, reliable products on top of AI models — choosing models, writing prompts, connecting data and tools, testing quality and running it all safely at scale., AI stackAI stack, n.: The layers of technology an AI product depends on, from chips and data centers at the bottom, through the models, up to the apps people use, each built on the layer below., chunkingchunking, n.: Cutting long documents into smaller pieces before storing them for AI search, so the system can find and hand the model just the relevant part instead of a whole manual., citationcitation, n.: A pointer from something an AI says to the source that supports it — a link, document or quoted passage — so people can check the claim for themselves., CohereCohere, prop. n.: A Canadian AI company, founded in Toronto in 2019, that builds language models and search tools for businesses and governments rather than consumer chatbots., context engineeringcontext engineering, n.: Managing an AI model's reading material: choosing, trimming and arranging the instructions, documents, tool results and memories it is given on every turn, so that it has what the task needs and little else., conversational AIconversational AI, n.: The technology that lets computers hold conversations with people in everyday language, by text or voice. It powers chatbots, voice assistants and automated customer-service lines., cosine similaritycosine similarity, n.: A score from −1 to 1 for how closely two lists of numbers point the same way, ignoring their size. It is the standard test of whether two texts mean similar things., document understandingdocument understanding, n.: Using AI to read documents such as invoices, forms, contracts and scientific papers the way a person does, taking in layout, tables and charts as well as words, to pull out the information needed., generative engine optimizationgenerative engine optimization, n.: Shaping web content so that AI chatbots and AI search features are more likely to find it, quote it and recommend it — the AI-era version of search engine optimization., glue on pizzaglue on pizza, n.: A notorious 2024 blunder in which Google's AI search summaries told people to add glue to pizza sauce, having taken an old joke from an online forum as genuine advice., GraphRAGGraphRAG, prop. n.: A way of letting AI answer questions about a big pile of documents by first turning them into a map of people, things and their connections, then using that map to find and summarize answers., groundinggrounding, n.: Connecting an AI's words to something real — trusted documents, a live search, images or the physical world — so that its answers rest on evidence rather than only on what it absorbed in training., hybrid searchhybrid search, n.: Searching two ways at once — by exact keywords and by meaning — and merging the results, so you catch precise matches like product codes as well as differently worded matches., information retrievalinformation retrieval, n.: The study and engineering of search: how to organize huge piles of documents so that the few that matter to someone's question can be found and ranked quickly. AI systems rely on it to look things up.