Paper breakdown

Chinchilla: how to split a compute budget between model size and training data

A technical reading of the 2022 paper that concluded large language models were undertrained — the three ways it estimated compute-optimal scaling, the Chinchilla-versus-Gopher test, and why later work revisits the recipe.

Author: HARBAN AI LABS
Published: 28 Sept 2026
Reading time: 5 min read

Read the original paper Cite

The paper

Training Compute-Optimal Large Language Models

Authors: Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, Tom Hennigan, Eric Noland, Katie Millican, George van den Driessche, Bogdan Damoc, Aurelia Guy, Simon Osindero, Karen Simonyan, Erich Elsen, Jack W. Rae, Oriol Vinyals, Laurent Sifre
Year: 2022
arXiv: 2203.15556source for arXiv (external site)
Checked against source: 28 Sept 2026

Contents

TL;DR

Given a fixed training compute budget, how large should a language model be, and how many tokens should it see? After training over 400 language models ranging from 70 million to over 16 billion parameters on 5 to 500 billion tokens, the authors conclude that model size and training tokens should be scaled equally: for every doubling of model size, the number of training tokens should also double. They tested the prediction by training Chinchilla, a 70B-parameter model that used the same compute as the 280B-parameter Gopher but four times more data, and report that it outperforms Gopher and several larger models across a wide range of tasks.

The problem

Training compute for a dense Transformer grows with both the number of parameters $N$ and the number of training tokens $D$ — roughly in proportion to their product. For a fixed budget $C$ , the question is how to split it:

N_{opt}(C),\, D_{opt}(C) = \operatorname*{arg\,min}_{N,\, D \;:\; \text{FLOPs}(N, D) = C} L(N, D)

where $L$ is the model's final pre-training loss.

Earlier work (Kaplan et al., 2020 (external site)) had concluded that compute-efficient training means training very large models on a relatively modest amount of data. The authors argue that the field's subsequent focus on growing models while keeping training data roughly constant left large models significantly undertrained.

They also point to a methodological cause. The earlier analysis used one fixed learning-rate schedule and token count for all models. Hoffmann et al. found that the best final loss comes from matching the length of the cosine learning-rate schedule to the number of training tokens. Losses read part-way through a longer schedule therefore overstate what a shorter, properly scheduled run would reach — which, the authors argue, contributed to the earlier conclusion that model size should grow faster than training data.

The method

The paper estimates the optimal split in three independent ways.

Fix model sizes, vary training length. Train a family of models, each for several different numbers of tokens. At every compute level, take the lowest loss any run achieved (the envelope of the training curves) and fit how the corresponding model size and token count grow with compute.
IsoFLOP profiles. For each of a set of fixed compute budgets, train models of different sizes — adjusting the token count so compute stays constant — and find the model size at the bottom of the resulting loss "valley".
Fit a parametric loss. Model the final loss of every run as

\hat{L}(N, D) = E + \frac{A}{N^{\alpha}} + \frac{B}{D^{\beta}}

where $E$ stands for the irreducible loss of natural text, the second term for the shortfall of a finite-sized model, and the third for finite training. The constants are fitted to all runs with a robust (Huber) loss, and $\hat{L}$ is then minimised under the compute constraint.

The three approaches give comparable predictions: parameters and tokens should grow at about the same rate as compute increases. All three imply that the large models of the time, Gopher included, were substantially over-sized for their compute budgets.

What the paper reports

The headline results, as reported by the authors in the abstract:

Claim	Result reported
Scaling experiments	Over 400 language models, from 70 million to over 16 billion parameters, trained on 5 to 500 billion tokens
Compute-optimal rule	Scale model size and training tokens equally: double the tokens for every doubling of model size
Chinchilla vs Gopher at equal compute	70B parameters and 4× more data than Gopher (280B)
Downstream evaluation	Chinchilla "uniformly and significantly outperforms" Gopher (280B), GPT-3 (175B), Jurassic-1 (178B) and Megatron-Turing NLG (530B) on a large range of tasks
MMLU	State-of-the-art average accuracy of 67.5%, described as greater than a 7% improvement over Gopher
Deployment	Substantially less compute for fine-tuning and inference, because the model is smaller

These are the authors' own measurements. The paper adds its own caution: because Chinchilla saw four times more data than Gopher, overlap between training and test data could inflate some language-modelling results, so the authors put more weight on benchmarks such as MMLU where that is less of a concern.

Why it mattered

In our reading, the paper made training tokens a first-class planning variable alongside parameter count. Later work engages with it directly:

LLaMA (Touvron et al., 2023 (external site)) builds on Hoffmann et al. but argues that their objective ignores the inference budget: if a model will be served at scale, a smaller model trained for longer ends up cheaper at inference, even if a larger model would have been cheaper to train to the same performance.
Beyond Chinchilla-Optimal (Sardana et al., 2023 (external site)) modifies the Chinchilla scaling laws to account for inference demand and concludes that models expecting large inference demand should be trained smaller and longer than Chinchilla-optimal.
A replication attempt (Besiroglu et al., 2024 (external site)) re-fitted the third approach using data reconstructed from the paper's plots. They report that the published estimates for that approach are inconsistent with the other two and that its confidence intervals are implausibly narrow — while their own re-fit agrees with the first two approaches.

Limitations and open questions

The authors list several limitations themselves:

Two large-scale data points. Only Chinchilla and Gopher were compared at large scale, with no tests at intermediate scales.
Power-law assumption. The analysis assumes the compute frontier follows a power law, but the authors observe some curvature at high budgets, which suggests they may still be overestimating the optimal size of large models.
Less than one epoch. Every run saw less than one pass over its data; the multi-epoch regime is left to future work.
Data quality. The authors speculate that scaling to larger datasets only helps when the data is high quality.

Two further points follow from the follow-up work above:

Training compute only. "Compute-optimal" here means optimal for training compute. Deployment costs can favour a different trade-off.
Fitted constants are setup-specific. The coefficients come from the paper's own data mixture and architecture, and the replication debate shows the third approach's estimates deserve care.

Reading notes for practitioners

Know which budget you are optimising. Chinchilla-optimal minimises training compute for a target loss. If the model will be served heavily, include inference cost.
Ask how many tokens a model saw. Parameter count alone does not describe a model; training tokens, and their quality, matter as much.
Use the rule for intuition, not coefficients. "Scale parameters and tokens together" is robust across the three approaches; exact constants should be re-fitted for your own setup.
Compare models at equal compute. The Chinchilla–Gopher comparison is persuasive because compute was held fixed. Comparisons that change size and compute together are much harder to interpret.

Scaling Large language models