Retrieval-augmented generation (RAG)

A RAG pipeline runs four steps for every query:

1. Embedding the query. The user's question is converted into a numerical vector that captures its semantic meaning. The same embedding model that indexed the source documents is used here so query and documents share the same vector space.

2. Retrieving relevant chunks. The system searches an index (typically a vector database, often combined with keyword search for hybrid retrieval) for the document chunks most semantically similar to the query. Modern systems usually return 5–20 chunks and rerank them with a more precise model.

3. Augmenting the prompt. The top chunks are inserted into the prompt as context, alongside the original question and a system instruction telling the model to ground its answer in the provided sources.

4. Generating the response. The LLM produces an answer using the retrieved context. Most production systems return source citations so users can verify claims against the original documents.

RAG solves three fundamental LLM limitations.

Knowledge cutoffs. Models only know what was in their training data through a cutoff date. RAG provides current information at query time, so a model can answer questions about events, prices, or documentation that did not exist when the model was trained.

Private data access. Models cannot answer from data that was never in their training corpus — internal documentation, customer records, proprietary research. RAG lets models answer from any document set you put behind a retrieval index.

Hallucination reduction. When models have to ground their output in retrieved text rather than relying on parametric memory, they fabricate less. Industry analysis shows that properly implemented RAG can reduce hallucination by up to 71% on grounded benchmarks.

RAG is also what powers AI search itself. Perplexity, Google AI Overviews, ChatGPT search, and Claude with web search are all RAG systems — the difference is that the document index is the open web rather than a private corpus.

RAG systems fail in two distinct ways, and you need separate metrics for each.

Retrieval quality. Did the system find the right documents? Measured by recall (the percentage of relevant documents that were retrieved) and precision (the percentage of retrieved documents that were relevant). Poor retrieval guarantees poor generation — no model can answer well from irrelevant context.

Generation quality. Given the retrieved context, did the model use it correctly? Measured by faithfulness (the response stays grounded in the retrieved sources without contradiction or fabrication) and answer relevance (the response actually addresses the question). The Vectara HHEM Leaderboard is the most widely referenced benchmark for grounded faithfulness.

Both metrics matter. A retrieval system with 90% recall feeding a generation step that hallucinates 30% of the time still produces a 27%-broken end product.

Five techniques have measured impact:

1. Hybrid retrieval. Combine dense vector search (semantic) with sparse keyword search (BM25 or similar). Each catches relevance signals the other misses.

2. Reranking. After initial retrieval returns 20–50 candidate chunks, rerank them with a cross-encoder model that scores each query-chunk pair more precisely. Reranking significantly improves precision at the top of the result list.

3. Chunking strategy. Source documents must be split into passages — too small and the model loses context, too large and irrelevant content dilutes the relevant signal. 100–300 word chunks with 10–20% overlap is the production-grade default for most domains.

4. Query rewriting. The user's literal query is often a poor retrieval query. Modern systems rewrite or expand queries before retrieval — converting "how do I do X" into structured search terms that match how documentation is actually written.

5. Evaluation pipelines. Build a ground-truth set of 50–500 query-answer pairs and run it on every change. RAG quality drifts silently as documents change and embeddings age. Continuous evaluation catches regressions before users do.