Research
Paper breakdowns
A breakdown is our technical reading of one influential paper: what problem it tackled, how the method works, what the authors report and where the evidence stops.
Numbers are the authors’ own reported results, quoted from the paper and checked against the source. We don’t re-run the experiments, and we say so.
Every breakdown covers
- TL;DR
- The problem
- The method
- What the paper reports
- Why it mattered
- Limitations
- Reading notes
Tagged “Large language models”
5 breakdowns of 6
Attention Is All You Need: the architecture behind modern language models
A technical reading of the 2017 paper that replaced recurrence with self-attention — what the Transformer does, why it trains faster, and what the paper did and did not show.
- Architectures
- Large language models
- Paper authors
- Vaswani et al.
- Paper year
- 2017
- arXiv identifier
- arXiv 1706.03762
Chain-of-Thought Prompting: eliciting step-by-step reasoning with a few examples
A technical reading of the 2022 paper showing that few-shot examples with written-out reasoning steps improve large language models on arithmetic, commonsense and symbolic tasks — and why the effect appeared only at large scale.
- Prompting
- Reasoning
- Large language models
- Paper authors
- Wei et al.
- Paper year
- 2022
- arXiv identifier
- arXiv 2201.11903
Chinchilla: how to split a compute budget between model size and training data
A technical reading of the 2022 paper that concluded large language models were undertrained — the three ways it estimated compute-optimal scaling, the Chinchilla-versus-Gopher test, and why later work revisits the recipe.
- Scaling
- Large language models
- Paper authors
- Hoffmann et al.
- Paper year
- 2022
- arXiv identifier
- arXiv 2203.15556
LoRA: fine-tuning large models by training small low-rank updates
A technical reading of the 2021 paper that freezes pre-trained weights and learns low-rank update matrices instead — the maths, why it adds no inference latency, what the authors report against full fine-tuning, and what remains open.
- Fine-tuning
- Large language models
- Paper authors
- Hu et al.
- Paper year
- 2021
- arXiv identifier
- arXiv 2106.09685
Retrieval-Augmented Generation: pairing a language model with a searchable memory
A technical reading of the 2020 paper that introduced RAG models — how a dense retriever and a sequence-to-sequence generator are trained together, the difference between RAG-Sequence and RAG-Token, and what the results do and do not show.
- Retrieval (RAG)
- Large language models
- Architectures
- Paper authors
- Lewis et al.
- Paper year
- 2020
- arXiv identifier
- arXiv 2005.11401
New breakdowns by email
Sign up for email updates on new paper breakdowns. No issue has been sent yet.