Retrieval-augmented generation (RAG): a complete guide

A language model only knows what it saw in training. Ask it about your company's Q3 numbers or a library released last week, and it either shrugs or makes something up. RAG fixes this by flipping the order: retrieve the relevant facts first, then let the model write its answer grounded in them. The original paper (Lewis et al., Facebook AI, 2020) paired dense passage retrieval (DPR) with a BART generator and hit state-of-the-art on open-domain QA benchmarks like Natural Questions and TriviaQA — and in production today, RAG cuts hallucinations by 30-50% and powers 60%+ of enterprise LLM applications.

See it work

The whole idea fits in one before/after. Watch what happens when the model has to reach past its training data.

Without RAG
Q: "What is the capital of Burkina Faso?"
A: [Model guesses from training data — might be outdated or wrong]

With RAG
Q: "What is the capital of Burkina Faso?"
Retrieved: "Burkina Faso's capital is Ouagadougou, located in the
            center of the country..."
A: "The capital of Burkina Faso is Ouagadougou. [Source: World Factbook]"

Same model, same question. The only difference is that the second run pasted a relevant passage into the prompt before asking. The answer is now correct, attributable, and verifiable — you can click the source.

The mental model

Think of RAG like a researcher with a library card. The researcher (the LLM) brings general knowledge and reasoning skills, but for specific facts they walk to the shelf, pull the right book (the retrieved documents), and quote from it. They don't try to memorize the whole library.

RAG separates knowledge storage from knowledge reasoning: the facts live in an updatable knowledge base, the LLM just reads and writes.

That separation is the entire payoff. You update facts by editing the library, not by retraining the brain.

How it works

Every RAG query runs through three phases — retrieve, augment, generate — sitting on top of a knowledge base you've prepared in advance.

1. Prepare the knowledge base (offline). Collect documents, clean the text, split them into retrievable chunks, attach metadata, embed each chunk into a vector, and load the vectors into an index.
2. Retrieve. Embed the user's query with the same model, then find the top-k most similar chunks by vector similarity.
3. Rerank (optional but recommended). Cast a wide net (say top-20), then use a cross-encoder to refine down to the best 3-5.
4. Augment. Paste those chunks into the prompt as context, with instructions to answer only from the sources and cite them.
5. Generate. The LLM writes an answer conditioned on both the query and the retrieved context, ideally with [Source N] citations.

Under the hood, retrieval ranks documents by cosine similarity — cosine(q, d) = (q · d) / (||q|| ||d||) — over embeddings in a shared vector space. Generation is formally a marginalization over retrieved documents, P(A|Q) = Σ_D P(A|Q,D) · P(D|Q): weight each candidate answer by how confident retrieval was in the document it came from. The 2020 paper offered two flavors of this — RAG-Sequence (same retrieved docs for the whole answer) and RAG-Token (different docs allowed per generated token).

Why it works

Not every part of the pipeline matters equally. Ranked by how much they move the needle:

The deeper principles: knowledge and reasoning are separated (update one without touching the other), answers are grounded in evidence (fewer hallucinations, real attribution), knowledge scales without retraining (add documents, not parameters), and dense embeddings retrieve by meaning — "how to reduce stress" finds passages on "anxiety relief" even with zero shared keywords, and can even bridge languages.

Where it shines

RAG is the default whenever answers must be factual, current, and traceable. Common task types: open-domain and closed-domain QA, document analysis, conversational assistants over a corpus, and any "answer from these sources" workflow.

The domain results are why it caught on:

Production examples span Notion AI and Glean (enterprise search), Intercom and Zendesk AI (support), UpToDate and BMJ Best Practice (medical), LawGeex and Kira Systems (legal), GitHub Copilot and Sourcegraph Cody (code), Semantic Scholar and Elicit (research), Bloomberg GPT and FinChat (finance), and Khan Academy and Duolingo (learning).

When to use it (and when not)

Reach for RAG when factual accuracy is critical, information changes frequently, citations and transparency are required, the knowledge base is large, or you need proprietary/domain-specific knowledge.

Skip it when the task is purely creative with no factual grounding needed, the knowledge is static and small enough to bake into a fine-tuned model, you need millisecond latency, or no suitable knowledge base exists.

Model fit. Any instruction-following generator works; lower the temperature (~0.1) for factual answers. The query and document embeddings must come from the same embedding model — mixing models silently wrecks retrieval.

Escalate to multi-hop or agentic variants when single-shot retrieval can't gather everything (see Advanced variants), and reach for hybrid RAG + fine-tuning when you need both domain fluency and current facts.

Variants and alternatives

RAG itself spans a maturity ladder (the naive → advanced → modular framing from the RAG survey literature):

And RAG competes (or combines) with other approaches:

Anatomy of a RAG pipeline

Five components, each a tuning surface.

1. Knowledge base preparation. Collect and clean documents, then chunk them. Chunking is the highest-leverage decision: fixed-size (simple, predictable, but breaks semantic units), semantic (split on paragraph/sentence boundaries, preserves coherence, variable size), recursive (split on a hierarchy: chapters → sections → paragraphs → sentences), or contextual (a 2024 technique — prepend each chunk with a document summary/title so it's self-contained). Adapt by document type: split code by function/class, legal docs by clause, papers by section. Attach metadata (source, page, section, author, date, type) for filtering, provenance, and citations.

2. Embedding and indexing. Turn chunks into vectors with an embedding model:

Store the vectors in a vector database — Pinecone (managed, serverless), Weaviate (open-source, GraphQL), Qdrant (Rust, fast), Milvus (production-scale), or Chroma (embedded, prototyping) — or bolt vectors onto an existing store via PostgreSQL + pgvector, Elasticsearch kNN, or Redis. Tune the index by similarity metric (cosine, Euclidean, dot product), index type (Flat for exact, HNSW or IVF for approximate), and quantization (PQ, SQ) to shrink memory.

3. Retrieval. Dense retrieval (semantic vector search) understands paraphrases and concepts; sparse retrieval (BM25, keyword) nails exact matches, rare terms, and proper nouns. Hybrid retrieval blends both and wins in most scenarios.

4. Reranking. Initial dense retrieval optimizes recall, not precision. A cross-encoder rescoring the top results lifts precision: retrieve top-20, rerank to the best 3-5, generate from those. Options include Cohere Rerank (commercial), MS MARCO cross-encoders (open-source), and ColBERT (late interaction).

5. Prompt augmentation and generation. Build the context block, instruct the model to use only the sources and cite them, and generate at low temperature.

The core loop

The whole pipeline, stripped to its essence:

def rag(query, vector_db, embed_model, llm):
    # 1. Retrieve: embed query, search, rerank
    q_vec = embed_model.encode(query)
    candidates = vector_db.query(q_vec, top_k=20, include_metadata=True)
    docs = rerank(query, candidates, top_k=3)   # cross-encoder

    # 2. Augment: build a grounded, citable prompt
    context = "\n\n".join(f"[Source {i+1}]: {d['text']}"
                          for i, d in enumerate(docs))
    prompt = (
        "Answer the question using only the context below. "
        "If the answer isn't there, say so. Cite sources as [Source N].\n\n"
        f"Context:\n{context}\n\nQuestion: {query}\n\nAnswer:"
    )

    # 3. Generate: low temperature for factual answers
    return llm(prompt, temperature=0.1)

A prompt template that grounds well

The instructions do real work — they're what turn pasted text into a faithful, cited answer.

You are a helpful assistant that answers from the provided documents.

Documents:
[Source 1 - "Python Documentation", updated 2024]: ...
[Source 2 - "PEP 701", published 2023]: ...

User Question: {query}

Instructions:
- Answer based only on the provided documents.
- If the information isn't there, say "The provided sources do not
  contain sufficient information to answer this question."
- If sources contradict, acknowledge it and present both with citations.
- Cite sources using [Source N] notation.

Answer:

Configuration

Sensible defaults for the knobs that matter most:

Use asymmetric embeddings (query: vs passage: prefixes) when queries are short and documents long — most RAG systems; use symmetric when both sides are similar in nature. Cache document embeddings and only recompute when content changes; embed queries fresh each time.

Implementation workflow

1. Ingest and chunk your corpus, choosing a chunking strategy per document type and attaching metadata.
2. Embed and index with one consistent embedding model into a vector database.
3. Build retrieval: start dense, add sparse for a hybrid, layer a reranker on top.
4. Manage the context window: reserve a response budget, then fit documents by truncation, summarization of retrieved context, or hierarchical retrieval (summaries first, then detailed chunks).
5. Write the augmentation prompt with source attribution and grounding instructions.
6. Add guardrails: handle no-results, low-confidence retrieval, and irrelevant hits (an LLM relevance check that filters before generation).
7. Evaluate, log failures, iterate.

Do and don't

Debugging

Read symptom → likely cause → fix:

• Right answer never appears → relevant chunk not retrieved → add hybrid (sparse) retrieval, expand the query, or raise initial top-k.
• Answer drifts from the sources → model leaning on parametric knowledge → strengthen grounding instructions, add a faithfulness check, or use CRAG-style correction.
• Context overflows the window → too many/too-long chunks → summarize retrieved context or switch to hierarchical retrieval.
• Near-duplicate chunks crowd out coverage → low diversity → apply Maximal Marginal Relevance (MMR) to balance relevance and diversity.
• Chunks lack context ("the experiment yielded a 23% improvement" — which experiment?) → use contextual chunking.
• Confidently wrong on out-of-scope queries → no abstain path → enforce a confidence threshold and an "I cannot find this" fallback.

Measuring it

Evaluate retrieval and generation separately, then end-to-end.

Retrieval metrics: Recall@K (fraction of relevant docs in the top-K), Precision@K (fraction of retrieved docs that are relevant), Mean Reciprocal Rank (MRR, rank of the first relevant doc), and NDCG (ranking quality weighted by position).

Generation metrics: Faithfulness/groundedness (claims supported by context), answer relevance (does it address the question), context relevance (was the retrieved context on-topic), and answer correctness (semantic similarity to ground truth).

The RAG Triad — context relevance, groundedness, answer relevance — is the standard end-to-end lens, and frameworks like RAGAS automate it (context precision, context recall, faithfulness, answer relevance):

from ragas import evaluate
from ragas.metrics import faithfulness, answer_relevance, context_precision

results = evaluate(
    dataset=eval_dataset,
    metrics=[faithfulness, answer_relevance, context_precision],
)

Benchmark against Natural Questions (open-domain QA), HotpotQA (multi-hop), FEVER (fact verification), MS MARCO (passage retrieval), and SQuAD (reading comprehension). Pair automated metrics with human review scoring accuracy, completeness, clarity, citation quality, and relevance — ground truth is often unavailable, and retrieval and generation errors compound, so isolating the failure point matters.

Advanced variants

Once naive RAG is working, these techniques attack specific weak spots:

• Query transformation. Query expansion generates paraphrases to widen recall. Query decomposition breaks a complex question into sub-questions retrieved separately. HyDE (Hypothetical Document Embeddings) generates a hypothetical answer, embeds that, and retrieves with it — the fake answer is document-like and often matches real documents better than the raw query.
• Multi-hop reasoning. Iteratively retrieve, answer partially, then retrieve again for the gap. "Who is the CEO of the company that acquired Instagram?" → hop 1 finds Facebook acquired Instagram → hop 2 finds Mark Zuckerberg.
• Self-RAG. The model decides whether to retrieve, generates, self-critiques the answer against the context, and revises if it spots unsupported claims.
• CRAG (Corrective RAG). Score each retrieved doc's relevance; if all scores are low, fall back to web search; otherwise keep only the high-scoring docs before generating.
• Graph RAG. Retrieve from a knowledge graph (entities and relationships) alongside text — strong for relationship-heavy and multi-entity queries in structured domains like medical and financial.
• Agentic RAG. RAG as one tool in an agent that plans, calls tools (retrieve, calculate, web search), and synthesizes across steps.
• Multimodal RAG. Retrieve and reason over images, tables, and charts (CLIP embeddings + a multimodal LLM) — product docs with diagrams, medical imaging plus reports.
• Contextual retrieval. Anthropic's 2024 technique of prepending document context to each chunk before embedding reduced failed retrievals by up to 67%.

Limitations

• Retrieval is the ceiling. Miss the right document and generation fails; semantic search still trips on different terminology, bad chunk boundaries, and rare queries.
• Context-window constraints. You trade breadth (more docs) against depth (longer passages); long docs get truncated and multi-document reasoning stays hard.
• The model may ignore context in favor of parametric knowledge, contradict retrieved facts, or fail to synthesize across documents.
• Cost and latency. Generation dominates cost; complex queries run 2-5 seconds.
• Update lag. Even an updatable knowledge base takes minutes to hours to ingest, embed, and index — risky for real-time domains.
• Evaluation is unsolved. No universal RAG benchmark; errors compound across components.
• Conflicting sources have no robust automated resolution — the best you can do is surface both and prefer authoritative sources.

Ecosystem and integration

Orchestration frameworks (LangChain, LlamaIndex) wire the pipeline together; vector databases (Pinecone, Weaviate, Qdrant, Milvus, Chroma) store the index; embedding models (OpenAI Ada-002, Cohere, BGE, E5, Sentence Transformers) and rerankers (Cohere Rerank, MS MARCO cross-encoders, ColBERT) handle retrieval quality; RAGAS and the RAG Triad handle evaluation.

RAG composes naturally with the techniques in its comparison table: fine-tuning for domain fluency layered under RAG for facts, long-context models for full-document analysis of retrieved docs, few-shot with dynamically retrieved examples, and chain-of-thought for reasoning over retrieved evidence. To transition into RAG from a plain LLM, start with a small curated knowledge base and naive retrieve-then-generate, then add reranking, hybrid search, and query transformation as quality demands. To transition out toward more capability, move up the ladder to modular and agentic systems.

Future directions

RAG is moving from simple pipelines toward agentic systems that plan multi-step retrievals, reason across modalities, self-correct, and personalize. Active frontiers: agentic RAG with tool use and planning; multimodal and cross-lingual retrieval; personalized and adaptive RAG that learns user preferences; real-time knowledge integration (streaming ingestion, incremental indexing); federated and private RAG with differential privacy and local embeddings; efficient architectures targeting ~10x cost reduction via compressed embeddings and faster ANN search; explainable RAG with claim-level attribution; standardized benchmark suites (a SuperGLUE for RAG); and hybrid RAG + fine-tuning as the production default for high-stakes domains.

Summary

• RAG = retrieve, then generate. It grounds answers in an external, updatable knowledge base instead of model parameters — separating knowledge storage from reasoning.
• The pipeline has five tunable parts: chunking, embedding/indexing, retrieval, reranking, and grounded generation. Retrieval quality is the hard ceiling on everything else.
• Reach for it when facts must be accurate, current, and cited; skip it for purely creative work, tiny static knowledge, or millisecond-latency needs.
• Production defaults: 256-512 token chunks with ~10% overlap, retrieve top-20 and rerank to 3-5, same embedding model for index and query, temperature ~0.1, and always cite sources.
• Measure with retrieval metrics (Recall@K, Precision@K, MRR, NDCG) and the RAG Triad (context relevance, groundedness, answer relevance) via RAGAS.
• Climb the ladder from naive to advanced (hybrid search, reranking, query transformation, contextual chunks) to modular (Self-RAG, CRAG, graph, agentic, multimodal) as your task demands.
• The payoff is real: 30-50% fewer hallucinations, 40-70% efficiency gains across support, search, and legal, and answers users can actually verify.