Paper breakdown

Retrieval-Augmented Generation: pairing a language model with a searchable memory

A technical reading of the 2020 paper that introduced RAG models — how a dense retriever and a sequence-to-sequence generator are trained together, the difference between RAG-Sequence and RAG-Token, and what the results do and do not show.

Author: HARBAN AI LABS
Published: 28 Sept 2026
Reading time: 4 min read

Read the original paper Cite

The paper

Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks

Authors: Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, Sebastian Riedel, Douwe Kiela
Year: 2020
Venue: NeurIPS 2020
arXiv: 2005.11401source for arXiv (external site)
Checked against source: 28 Sept 2026

Contents

TL;DR

The paper combines two kinds of memory in one model. Parametric memory is what a pre-trained sequence-to-sequence model stores in its weights; non-parametric memory is a dense vector index of Wikipedia, searched by a neural retriever. For each input the model retrieves passages and generates its output conditioned on them, and the retriever and generator are fine-tuned together. The authors report state-of-the-art results on three open-domain question-answering tasks, and generations that are more specific, diverse and factual than a parametric-only baseline.

The problem

Large pre-trained language models store a surprising amount of factual knowledge in their parameters, but the authors spell out the drawbacks of keeping knowledge there: it is hard to access precisely, hard to update, and gives no provenance for an answer. On knowledge-intensive tasks — ones a person could not reasonably do without looking something up — such models trailed task-specific systems.

Earlier hybrid models such as REALM (Guu et al., 2020 (external site)) had already paired a language model with a differentiable retriever, but only for extractive question answering, where the answer is a span copied out of a document. The question here is whether the same idea works for general-purpose generation.

The method

Components

Retriever. Based on Dense Passage Retrieval (Karpukhin et al., 2020 (external site)): a bi-encoder in which one BERT encoder embeds the query and another embeds each passage. Relevance is the inner product of the two embeddings, and the top- $k$ passages are found with maximum inner product search over a pre-built index of Wikipedia.
Generator. BART, a pre-trained sequence-to-sequence Transformer. The retrieved passage is simply concatenated with the input.
Training. The retrieved passage is treated as a latent variable. Training minimises the negative marginal log-likelihood of the target output, with no supervision about which passage should have been retrieved. Only the query encoder and the generator are fine-tuned; the passage encoder and index stay fixed, which avoids the cost of rebuilding the index during training.

Two ways to use the retrieved passages

With a retriever $p_\eta(z \mid x)$ over passages $z$ and a generator $p_\theta$ , the paper defines two models.

RAG-Sequence uses the same passage for the whole output, then sums over the top- $k$ passages:

p(y \mid x) \approx \sum_{z \in \text{top-}k} p_\eta(z \mid x) \prod_{i} p_\theta(y_i \mid x, z, y_{1:i-1})

RAG-Token can draw on a different passage for each generated token:

p(y \mid x) \approx \prod_{i} \sum_{z \in \text{top-}k} p_\eta(z \mid x)\, p_\theta(y_i \mid x, z, y_{1:i-1})

RAG-Token lets one answer combine facts from several passages; RAG-Sequence ties each candidate output to a single passage. For classification, where the output is a single token, the two are equivalent.

What the paper reports

The headline results, as reported by the authors in the abstract:

Task type	Result reported
Open-domain question answering	State of the art on three open-domain QA tasks, outperforming parametric seq2seq models and task-specific retrieve-and-extract architectures
Knowledge-intensive language generation	More specific, diverse and factual language than a state-of-the-art parametric-only seq2seq baseline

The body of the paper evaluates open-domain question answering on four datasets: Natural Questions, TriviaQA, WebQuestions and CuratedTrec. The three state-of-the-art results in the abstract are on Natural Questions, WebQuestions and CuratedTrec; for TriviaQA, the authors claim state of the art only on the split comparable with T5. The paper also covers MS MARCO, Jeopardy question generation and FEVER fact verification. Two further findings are worth knowing:

Swapping the index updates knowledge. When the authors replaced the Wikipedia index with an older snapshot, the model's answers about world leaders tracked whichever index was loaded — its knowledge changed without retraining.
Learned retrieval helps, mostly. Freezing the retriever during training hurt results on every task, but a word-overlap retriever (BM25) did best on FEVER, whose claims are heavily centred on named entities.

These are the authors' own measurements under their own setup.

Why it mattered

In our reading, the durable idea is a separation of concerns: knowledge lives in an index that can be inspected and replaced, and the model learns to use what the index returns. Two verifiable follow-ups:

The authors released their code as part of Hugging Face Transformers, which documents RAG (external site) with separate retriever and generation classes and links back to this paper.
Atlas (Izacard et al., 2022 (external site)), with this paper's first author among its co-authors, pre-trains a retrieval-augmented model for few-shot learning and likewise shows that its document index can easily be updated.

Limitations and open questions

Retrieval can collapse. In preliminary experiments on some tasks, such as story generation, the retriever learned to return the same passages whatever the input, and the generator learned to ignore them — at which point the model behaved like plain BART.
A fixed passage encoder. Keeping the index fixed made training practical, but the passage side never adapts to the task.
One curated source. Every experiment uses a single Wikipedia snapshot. In their broader-impact statement the authors note that no external knowledge source will be entirely factual or free of bias.
Generation is not citation. Nothing in the method forces the output to be supported by the retrieved passages. The paper's evidence for factuality comes from aggregate evaluation, not a per-answer guarantee.

Reading notes for practitioners

Measure retrieval on its own. If the right passage is not in the top- $k$ , generation cannot fix it. Check retrieval recall on your own questions before tuning prompts.
Test that context is actually used. The retrieval-collapse finding is a reminder that a model can learn to disregard retrieved text. Alter a retrieved passage deliberately and confirm the answer changes.
Keep a lexical baseline. BM25 beat the dense retriever on an entity-heavy task in this very paper; hybrid retrieval is often worth testing.
Index freshness is a feature — if you monitor it. Knowledge can change without retraining, but only if someone owns keeping the index current.
Provenance takes deliberate work. Showing sources to users means tracking which passages were used and checking the answer against them.

Retrieval (RAG)Large language models Architectures

The paper

Related breakdowns