A practical checklist for shipping a RAG system

Two systems, not one

Retrieval-augmented generation, as introduced by Lewis et al., paired a pre-trained sequence-to-sequence model with a dense vector index of Wikipedia accessed through a neural retriever (arXiv:2005.11401 (external site); see our paper breakdown). Production systems today are broader, but the shape is the same: something retrieves passages at query time, and a model writes an answer conditioned on them.

Treat those as two systems with separate tests. If retrieval does not return the passage that contains the answer, no amount of prompt work will fix the output. Most of this checklist is about the retrieval half, because retrieval failures cap everything downstream and retrieval is the cheaper half to measure.

1. Build a retrieval evaluation set before you tune anything

You cannot compare chunking strategies, embedding models or rerankers without a fixed set of questions with known answers.

• Use real questions. Pull them from search logs, support tickets or interviews with the people who will use the system. Synthetic questions are a useful supplement, not a substitute.
• Label relevant sources. For each question, record which documents (and ideally which section) contain the answer. Label at document-and-section level rather than chunk level, so your labels survive re-chunking.
• Include unanswerable questions. The system should decline when the corpus does not contain the answer; you can only test that if such questions are in the set.
• Keep it versioned alongside the code, and grow it whenever a production failure teaches you something.

Two metrics cover most needs. Recall@k is the share of relevant documents that appear in the top k results; mean reciprocal rank (MRR) rewards putting the first relevant result near the top.

def recall_at_k(retrieved: list[str], relevant: set[str], k: int) -> float:
    """Share of the relevant documents that appear in the top k results."""
    return len(set(retrieved[:k]) & relevant) / len(relevant)


def reciprocal_rank(retrieved: list[str], relevant: set[str]) -> float:
    """1 / rank of the first relevant result, or 0 if none was retrieved."""
    for rank, doc_id in enumerate(retrieved, start=1):
        if doc_id in relevant:
            return 1.0 / rank
    return 0.0


def evaluate(runs: dict[str, list[str]], qrels: dict[str, set[str]], k: int = 5) -> dict[str, float]:
    """Average both metrics over queries that have at least one relevant document."""
    answerable = [q for q, relevant in qrels.items() if relevant]
    n = len(answerable)
    return {
        f"recall@{k}": sum(recall_at_k(runs[q], qrels[q], k) for q in answerable) / n,
        "mrr": sum(reciprocal_rank(runs[q], qrels[q]) for q in answerable) / n,
    }

Here runs maps each query ID to the ranked document IDs your retriever returned, and qrels maps it to the set of relevant IDs. Unanswerable queries are excluded from these averages and scored separately, on whether the system declines. Pick the k that matches how many passages you actually put in the prompt.

2. Chunk by structure, then measure

• Split on the document's own structure first — headings, sections, list items, table rows — and fall back to fixed-size windows only inside sections that are too long.
• Keep metadata with every chunk: document ID, title, heading path, source URL, last-updated date and access-control information. You will need all of it later in this checklist.
• Give chunks context. A chunk that says "the limit is 30 days" is useless without knowing which policy it belongs to. Prepending the document title and heading path to the text you embed is a cheap fix.
• Handle tables and code separately. Splitting a table mid-row or a function mid-body produces chunks that match queries but cannot answer them.
• Consider small-to-large retrieval: match on small chunks for precision, then give the model the enclosing section for context.

Treat chunk size and overlap as hyperparameters. Choose them by comparing recall@k on your evaluation set, not by convention.

3. Use hybrid search, and fuse ranks rather than scores

Dense embeddings are good at paraphrase; lexical search such as BM25 is good at exact identifiers, error codes, product names and rare terms. Real queries need both.

Their scores are on different scales, so adding them directly is fragile. Fuse the rankings instead. Reciprocal Rank Fusion (Cormack, Clarke and Büttcher, SIGIR 2009) scores each document by summing 1 / (k + rank) across the input rankings; the authors fixed k = 60 in a pilot, reporting it near-optimal and the choice not critical (paper, PDF (external site)).

from collections import defaultdict


def reciprocal_rank_fusion(rankings: list[list[str]], k: int = 60) -> list[str]:
    """Fuse several ranked lists of document IDs (best first) into one."""
    scores: dict[str, float] = defaultdict(float)
    for ranking in rankings:
        for rank, doc_id in enumerate(ranking, start=1):
            scores[doc_id] += 1.0 / (k + rank)
    return sorted(scores, key=scores.__getitem__, reverse=True)


bm25_ids = ["doc-7", "doc-2", "doc-9"]
dense_ids = ["doc-2", "doc-4", "doc-7"]
print(reciprocal_rank_fusion([bm25_ids, dense_ids]))
# ['doc-2', 'doc-7', 'doc-4', 'doc-9']

A reranking model applied to the top few dozen fused results can help further. Keep it only if it improves recall at your prompt's k on your evaluation set by enough to justify the added latency.

4. Make citations verifiable

An answer with citations is only trustworthy if the citations can be checked — by the user and by your code.

• Give every passage a stable ID in the prompt and ask the model to cite IDs in square brackets next to the claims they support. An answer may group several IDs in one bracket ([doc-2, doc-5]), so the check below reads every ID in every bracket.
• Validate after generation. Any cited ID that was not in the retrieved context is a fabrication; flag or regenerate.

import re

BRACKETS = re.compile(r"\[([^\[\]]+)\]")
DOC_ID = re.compile(r"doc-[A-Za-z0-9-]+")


def unsupported_citations(answer: str, retrieved_ids: set[str]) -> list[str]:
    """Cited IDs that were not in the context the model was given.

    Checks every ID inside every bracket, so grouped citations such as
    [doc-2, doc-3] are validated as well as single ones.
    """
    cited = [cid for group in BRACKETS.findall(answer) for cid in DOC_ID.findall(group)]
    return [cid for cid in cited if cid not in retrieved_ids]


unsupported_citations("A [doc-2] B [doc-99] C [doc-2, doc-3]", {"doc-2"})
# ['doc-99', 'doc-3']

• Link citations to the source passage, not just the document, and show its last-updated date.
• Decline when retrieval is empty or weak. If nothing relevant was retrieved, say so instead of letting the model answer from memory.
• Order context deliberately. Models do not use long inputs evenly: one study found performance often highest when relevant information is at the beginning or end of the context and degraded in the middle (Liu et al., 2023 (external site)). Put the strongest passages first and keep the context short.

5. Enforce permissions at retrieval time

If a user cannot open a document in the source system, its text must never reach the prompt for that user. Once it is in the context, you cannot reliably stop the model from repeating it.

• Filter in the retrieval query using access-control metadata stored on each chunk. Do not retrieve broadly and filter the answer afterwards.
• Sync permissions, not just content. When access changes in the source system, update the index metadata on the same schedule as content changes.
• Test it. Run part of your evaluation set as users with restricted access, include "canary" documents they must not see, and assert that none of those IDs are ever retrieved.
• Scope caches. Cached retrievals or answers must be keyed by the user's permission set, never shared across it.

6. Plan for freshness and deletion

• Index incrementally from change events or scheduled diffs, and store a last-updated date on every chunk.
• Propagate deletions. When a document is deleted or withdrawn, remove its chunks and verify it can no longer be retrieved. This matters for correctness, and often for legal and privacy obligations.
• Version the index. Vectors produced by different embedding models are not comparable; changing the embedding model means re-embedding the corpus and swapping indexes, not mixing them.
• Re-run the retrieval evaluation on every rebuild and block the swap on a regression.

7. Evaluate generation separately

Once retrieval is measured, evaluate the answers:

• Groundedness: is every claim supported by a cited passage?
• Correctness: does the answer match a reference answer for the question?
• Abstention: does the system decline on unanswerable questions, and avoid declining on answerable ones?

If you use a model as a judge, check its verdicts against human labels on a sample before trusting it at scale, and re-check whenever you change the judge model or prompt. For how to report the results honestly — sample sizes and confidence intervals included — see How to read an AI benchmark table.

8. Monitor in production

Log enough per request to reconstruct what happened:

• the query, retrieved IDs with scores, and the index version;
• model, prompt and retriever versions;
• latency per stage and token counts;
• cited IDs and the citation-validation result;
• whether the system declined, and any user feedback.

Watch the rates that signal drift: empty or low-score retrievals, citation-validation failures, abstentions, latency and cost per query. Review a sample of real conversations regularly and turn failures into new evaluation cases. These logs contain user queries, so give them a retention period and access controls from day one.

The checklist

• A versioned retrieval evaluation set with real and unanswerable questions
• Recall@k and MRR tracked for every retrieval change
• Structure-aware chunks carrying title, heading path, URL, date and access metadata
• Hybrid lexical + dense retrieval with rank fusion; reranker only if it earns its latency
• Stable passage IDs, citation validation and a decline path
• Permission filtering inside the retrieval query, tested with canary documents
• Incremental indexing, verified deletions and versioned indexes
• Separate groundedness, correctness and abstention evaluation
• Per-request logging, drift dashboards and a feedback loop into the evaluation set