contextual retrieval

A retrieval-augmented generationretrieval-augmented generation, n.: Giving an AI an open-book exam: before answering, it looks up relevant passages in a set of documents and bases its reply on them, often citing where each point came from. indexing technique, described by Anthropic in 2024, for chunkschunking, n.: Cutting long documents into smaller pieces before storing them for AI search, so the system can find and hand the model just the relevant part instead of a whole manual. that lose their meaning once cut from their document. A language model reads the whole document and writes a sentence or two placing each chunk — which document, section, entity and period it concerns — and this note is attached to the front of the chunk before it is embedded (contextual embeddings) and before it enters the keyword index (contextual BM25). A chunk saying only that a deadline was moved back two weeks, for instance, gains the name of the project and the date of the change. In its introducing report, adding context cut the top-20 retrieval failure rate by 35% with embeddings alone, by 49% when combined with contextual BM25BM25, prop. n.: A decades-old formula that search engines use to score how well a document matches a query, rewarding rare query words and repeated matches — with diminishing returns — and correcting for long documents., and by 67% when rerankingreranking, n.: A second, more careful pass over search results: a fast method grabs a few hundred candidates, then a slower, smarter model reorders them so the best come first. was added. The cost is one model call per chunk at indexing time, which prompt cachingprompt caching, n.: Saving the work a model did reading the start of a prompt, so later requests that begin the same way can skip that reading and come back faster and cheaper. of the shared document keeps affordable.

With contextual retrieval, each chunk now opens with a line naming the company, the filing and the quarter.