Retrieval-Augmented Generation (RAG)

Retrieval-augmented generation (RAG) is a technique that connects a language model to an external collection of documents through a retriever, so the model writes each answer conditioned on passages fetched for that specific query instead of relying only on what is stored in its weights. The name comes from a 2020 paper by Patrick Lewis and colleagues at Facebook AI Research, University College London and New York University, which paired a pre-trained sequence-to-sequence model (the 'parametric memory') with a dense vector index of Wikipedia (the 'non-parametric memory') and found the combination produced 'more specific, diverse and factual language' than the model alone (Lewis et al.).

This entry describes the technique. For how ChatGPT, Perplexity and Google's AI features apply it to web search, see RAG.

A RAG system has an offline part and a query-time part. Offline, documents are split into chunks of a few hundred tokens, each chunk is converted into a vector with an embeddings model, and the vectors are stored in an index. Most production systems also build a keyword index so they can run hybrid search. At query time, the system retrieves candidate chunks, reorders them with a reranking model, places the top results into the model's context window, and generates an answer that cites them.

Chunking is where many systems lose accuracy. Anthropic's September 2024 write-up on Contextual Retrieval gives a typical failure: a chunk that says 'the company's revenue grew by 3%' cannot be matched to a question about a named company and quarter, because the name and date sat in an earlier paragraph. Prepending 50 to 100 tokens of chunk-specific context before embedding and keyword indexing cut top-20 retrieval failures by 49% in Anthropic's tests, and by 67% when combined with reranking (Anthropic).

Several variants build on the basic design. Agentic RAG lets the model decide what to search for, read the results and search again, which is how deep research products work. Adaptive retrieval decides per query whether retrieval is needed at all and how much. Graph-based RAG retrieves over a knowledge graph of entities and relationships rather than over loose text chunks.

RAG is often compared with two alternatives. Long context means putting the whole corpus into the prompt; Anthropic noted that a knowledge base under about 200,000 tokens can simply be included, and current Claude models accept 1 million tokens. That works for a manual, not for the web. Fine-tuning changes a model's behavior and style, but it is a poor way to keep facts current, because every update needs another training run.

For content owners, each stage of a RAG pipeline is a filter. A page has to be indexed, chunked into passages that still make sense on their own, matched by both keyword and vector retrieval, and scored highly by a reranker before the model sees it. Retrieval evaluation is how engineering teams test each of those stages, and the same logic explains why some well-ranking pages never appear in AI answers.