Research
A breakdown is our technical reading of one influential paper: what problem it tackled, how the method works, what the authors report and where the evidence stops.
Numbers are the authors’ own reported results, quoted from the paper and checked against the source. We don’t re-run the experiments, and we say so.
Every breakdown covers
A technical reading of the 2017 paper that replaced recurrence with self-attention — what the Transformer does, why it trains faster, and what the paper did and did not show.
The paper
Attention Is All You Need
6 breakdowns
A technical reading of the 2017 paper that replaced recurrence with self-attention — what the Transformer does, why it trains faster, and what the paper did and did not show.
A technical reading of the 2022 paper showing that few-shot examples with written-out reasoning steps improve large language models on arithmetic, commonsense and symbolic tasks — and why the effect appeared only at large scale.
A technical reading of the 2022 paper that concluded large language models were undertrained — the three ways it estimated compute-optimal scaling, the Chinchilla-versus-Gopher test, and why later work revisits the recipe.
A technical reading of the 2021 paper that freezes pre-trained weights and learns low-rank update matrices instead — the maths, why it adds no inference latency, what the authors report against full fine-tuning, and what remains open.
A technical reading of the 2022 paper that had language models alternate between written reasoning and tool actions — how the thought–action–observation loop works, what it improved on question answering and interactive tasks, and where it failed.
A technical reading of the 2020 paper that introduced RAG models — how a dense retriever and a sequence-to-sequence generator are trained together, the difference between RAG-Sequence and RAG-Token, and what the results do and do not show.
Sign up for email updates on new paper breakdowns. No issue has been sent yet.