Chain-of-Thought Prompting: eliciting step-by-step reasoning with a few examples

TL;DR

Instead of showing a model only question–answer pairs in a few-shot prompt, show it question–reasoning–answer triples. The paper calls the written-out reasoning a chain of thought. With no fine-tuning at all, this change improved performance on arithmetic, commonsense and symbolic reasoning benchmarks — but only in sufficiently large models. The headline result, as the authors report it: a 540B-parameter model prompted with just eight chain-of-thought exemplars reached state-of-the-art accuracy on the GSM8K benchmark of math word problems.

The problem

By early 2022, making language models bigger had improved many tasks, but multi-step reasoning — math word problems, questions that need several inferential hops, simple symbolic manipulation — improved much less under standard few-shot prompting. Two earlier lines of work pointed in useful directions, each with a cost:

• Training with rationales. Models trained or fine-tuned to write natural-language intermediate steps before the answer did better on arithmetic, but building large sets of high-quality rationales is expensive.
• Few-shot prompting. Prompting a model with a handful of input–output examples avoids training entirely, but on reasoning tasks it worked poorly and often did not improve much as models grew.

The authors asked whether the two ideas could be combined: put the rationales into the few-shot examples themselves.

The method

A standard few-shot prompt contains exemplars of the form ⟨input, output⟩. A chain-of-thought prompt uses ⟨input, chain of thought, output⟩, where the chain of thought is a short sequence of natural-language steps that leads to the answer. The model sees a few such exemplars followed by a new question and continues the pattern: it writes its own reasoning steps, then an answer.

Key properties of the setup:

• No training. Every result comes from prompting off-the-shelf models; no weights change.
• Hand-written exemplars. The authors wrote one small set of exemplars with reasoning chains and reused it across most of the arithmetic benchmarks, rather than engineering a prompt per task.
• Greedy decoding. The model generates a single chain per question, and the final answer is read from the end of it.

The paper evaluates three families of tasks: arithmetic (GSM8K, SVAMP, ASDiv, AQuA and MAWPS), commonsense reasoning (including CSQA and StrategyQA), and symbolic reasoning (toy tasks such as last-letter concatenation and tracking whether a coin is still heads up after a series of flips).

Ablations: why does it help?

To test whether the benefit really comes from the reasoning, the authors compared the full method against three variants:

• Equation only — the model writes just a mathematical equation before the answer.
• Variable compute only — the model writes a string of dots, to test whether spending extra tokens is what helps.
• Reasoning after the answer — the chain comes after the answer, so the answer cannot depend on it.

None of these matched chain-of-thought prompting on GSM8K. The authors read this as evidence that expressing intermediate steps in natural language matters — not just extra computation, and not just activating knowledge the model already had.

What the paper reports

The headline results, as reported by the authors in the abstract:

The abstract gives no accuracy figures; per-benchmark numbers are in the paper's tables. These are the authors' own measurements under their own prompts and decoding settings. Prompting results are sensitive to which exemplars are used and how they are written — the paper tests this itself with exemplars written by different annotators — so they are best compared within the paper rather than across papers.

Why it mattered

The result needed no training: a change to the prompt alone changed measured performance, and the gain grew with model scale. Several well-documented follow-ups build directly on it:

• Self-consistency (Wang et al., 2022 (external site)) replaces greedy decoding with sampling several reasoning paths and choosing the most consistent final answer.
• Zero-shot chain of thought (Kojima et al., 2022 (external site)) elicits step-by-step reasoning without any exemplars, by adding "Let's think step by step" before each answer.
• ReAct (Yao et al., 2022 (external site)) interleaves reasoning traces with actions such as searching Wikipedia — see our ReAct breakdown.
• Instruction fine-tuning (Chung et al., 2022 (external site)) studies fine-tuning on chain-of-thought data as part of scaling instruction-tuned models.

Limitations and open questions

The authors state several limitations themselves:

• Is it reasoning? A chain of thought resembles how people reason, but the paper explicitly leaves open whether the network is actually reasoning.
• No guarantee of correct chains. A generated chain can be wrong and still land on the right answer, or read plausibly and be wrong. The paper's own error analysis found both.
• Scale dependence. Gains appeared only in large models; smaller models produced fluent but illogical chains. The authors note this makes the method costly to serve.
• Prompt sensitivity. Results vary with the exemplars and their author, although chain-of-thought prompting beat standard prompting across the annotators tested.
• Faithfulness. The paper notes that fully characterising the computation behind an answer remains open — a written chain is not proof of how the answer was produced.

Reading notes for practitioners

• Treat the chain as output, not as an explanation. It helps with debugging and review, but the paper does not show that it faithfully reflects how the model reached its answer.
• Score the final answer separately. Extract and check the answer itself; a well-written chain does not imply a correct result.
• Test on the model you will deploy. The benefit depended on scale in this paper; do not assume it transfers to a smaller or different model.
• Budget for longer outputs. Every reasoning token is generated and paid for. Measure latency and cost alongside accuracy.
• Version your exemplars. They are part of the system. Check how sensitive results are to rewording them, as the authors did.