How to read an AI benchmark table without fooling yourself

The short version

A benchmark score is a measurement of a system — model, prompt, sampling settings, tools, harness and grader — on a specific version of a dataset, reported by someone. A table flattens all of that into one cell. Before you compare two cells, recover what was flattened.

Our benchmark methodology explains how we label every number on this site. This piece is the reader's side of the same discipline: eight questions, in the order we ask them, with the evidence behind each.

1. Who produced the number?

Results come from three kinds of source, and they deserve different levels of trust:

• Vendor-reported — published by the organisation that built the model, usually in a launch post or model card. The vendor chose which benchmarks to report and which settings to use, and nobody else has necessarily reproduced the run.
• Third-party — measured by an independent evaluator with its own harness. More comparable across models, but only within that evaluator's setup.
• Internal — measured by us, with published settings and logs. We have not published internal results yet; when we do, they will carry their run logs.

Vendor numbers are not wrong by default. They are simply not a controlled comparison. Two vendors reporting on the same benchmark can differ in prompt format, number of samples, reasoning budget and tool access, and neither launch post may say so. The practical rule: do not rank vendor-reported numbers from different vendors as if one lab had measured them all. On our benchmark pages every result carries a source badge for exactly this reason.

2. Which benchmark, exactly?

Benchmark names hide variants and versions.

• Subsets. SWE-bench and SWE-bench Verified are different test sets. Verified is a 500-task subset in which human annotators reviewed each task for a clear problem statement, correct tests and solvability (SWE-bench (external site)). A score on one tells you little about the other.
• Versions. Terminal-Bench 2.1 was released to fix issues in 28 of the 89 tasks in Terminal-Bench 2.0 (Terminal-Bench (external site)). Scores on 2.0 and 2.1 are measurements on different tests, and the public leaderboard has since moved to version 4.0 (Terminal-Bench leaderboard (external site)); see our Terminal-Bench page.
• Time windows. LiveCodeBench annotates problems with release dates so that models can be evaluated on problems from a chosen period (LiveCodeBench (external site)). Two LiveCodeBench scores are only comparable if they cover the same date window.
• Editions. An "AIME" score refers to a specific year's exam papers — AIME 2024 and AIME 2025 are different problem sets.

If a table does not state the subset, version or window, treat the column as ambiguous.

3. Under what settings?

The same model on the same benchmark can produce very different numbers depending on how it was run:

• Attempts and aggregation. pass@k is the chance that at least one of k attempts succeeds; a score reported as pass@1 and one reported with several attempts (or a majority vote over many samples) are different quantities.
• Reasoning budget. Where a model offers a configurable reasoning budget or effort level, a result at the maximum setting says little about the setting you will actually pay for.
• Prompting and tools. Zero-shot versus few-shot, chain-of-thought or not, code execution or web search allowed or not.
• Cost. ARC Prize publishes evaluation costs alongside scores and uses retail pricing to assess cost efficiency (ARC Prize policy (external site)). A higher score at ten times the cost is a different product decision, not a better model in the abstract.

When a source does not state its settings, we record "Conditions not stated" rather than guess defaults. You should do the same in your own notes.

4. Is the benchmark still separating models?

Benchmarks age. Two failure modes matter most.

Label errors. If some answer keys are wrong, even a model that answers every question correctly cannot score 100%, and differences near the top are partly noise. An audit of MMLU estimated that 6.49% of its questions contain errors, and that 57% of the analysed Virology questions do (Gema et al., 2024 (external site)). For GPQA Diamond, a 198-question set, Greg Burnham noted that state-of-the-art scores cluster in a narrow band around 83%. Extrapolating aggressively from a handful of inspected questions, he arrived at an invalid-question rate of about 8% — while calling it a stretch to apply that rate to the whole set — and concluded that at least 90% of the benchmark is likely valid. The post is an informal analysis in Epoch AI's Gradient Updates series, which represents its author's views rather than Epoch AI's (Burnham, Epoch AI Gradient Updates, May 2025 (external site)).

Saturation. When the leading models all sit near the ceiling, the benchmark has stopped telling them apart. Each of our benchmark pages carries a dated health status — active, saturated, flawed or retired — with the source for the judgement. MMLU, for example, is marked flawed.

5. Could the model have seen the test?

Public test sets can leak into training data, and a memorised answer looks exactly like a reasoned one in a score table. One way to detect this is to generate fresh but equivalent versions of the problems and compare. Researchers who rewrote 41.2% of the MATH benchmark in functional form, so that new snapshots of each problem can be generated, reported "reasoning gaps" — the percentage drop in accuracy from the static to the functional version — of 58.35% to 80.31% across the models they tested (Srivastava et al., 2024 (external site)). They measured those gaps with a simple few-shot prompting strategy and expect them to be smaller with more sophisticated prompting, so treat the range as an upper bound on what better prompting would show. See our MATH page.

Mitigations you can look for in a benchmark's design: date windows (LiveCodeBench), held-out private test sets run by the maintainers (ARC Prize's semi-private set, see ARC-AGI-2), and freshly written or regenerated items.

6. What does the grader actually check?

A score is only as good as its tests. When the HumanEval tests were extended roughly 80-fold, pass@k fell by up to 19.3–28.9%, and the ranking changed: two models that trailed ChatGPT on HumanEval overtook it on the stricter version (Liu et al., 2023 (external site)).

Agentic coding benchmarks have the same problem at larger scale. An analysis of SWE-bench found that 32.67% of one agent's successful patches had the solution given in the issue report or its comments, and that 31.08% of its passing patches were suspect because the tests were too weak to verify them; after filtering out those issues, that agent's resolution rate fell from 12.47% to 3.97% (Aleithan et al., 2024 (external site)).

Ask: does passing the grader imply the task was actually done?

7. Is the difference bigger than the noise?

Every benchmark is a sample of possible questions, so every score has sampling error. For accuracy on n independent questions, the standard error is roughly √(p(1−p)/n). A quick check:

import math

def accuracy_ci(correct: int, n: int, z: float = 1.96) -> tuple[float, float, float]:
    """Accuracy with a normal-approximation confidence interval (95% by default)."""
    p = correct / n
    se = math.sqrt(p * (1 - p) / n)
    return p, max(0.0, p - z * se), min(1.0, p + z * se)

p, lo, hi = accuracy_ci(158, 198)
print(f"{p:.3f} [{lo:.3f}, {hi:.3f}]")  # 0.798 [0.742, 0.854]

On a 198-question benchmark, a model scoring about 80% has a 95% confidence interval of roughly ±5.6 percentage points — before counting run-to-run variation from sampling. A three-point gap between two single runs on such a set is not, on its own, evidence of a real difference.

Evan Miller's "Adding Error Bars to Evals" sets out practical recommendations: compute standard errors with the Central Limit Theorem, use clustered standard errors when questions come in related groups, and compare two models on question-level paired differences rather than on their headline averages (Miller, 2024 (external site)). When a source publishes an interval, we show it; when it doesn't, the absence is itself information.

8. Which harness produced it?

For agent benchmarks, the number belongs to a model and the scaffold around it: the system prompt, the tools exposed, the retry policy, the context management. Terminal-Bench's leaderboard lists an agent column next to the model column and draws 95% confidence intervals for each entry (Terminal-Bench leaderboard (external site)). Comparing a model inside a vendor's own agent with a different model inside a generic one mixes two variables.

Reliability is a separate axis again. τ-bench introduced pass^k — the chance that all k independent trials of a task succeed, averaged across tasks — because a system that sometimes succeeds is not the same as one that reliably does (Yao et al., 2024 (external site)). Note that our τ-bench page records a documented grading flaw; the metric idea is sound even where a specific implementation was not.

A checklist to paste into a review

How we apply this

On HARBAN pages, every result carries its source type, the benchmark variant and settings as stated by the source, and a confidence interval where the source published one. We never chart results from different comparability groups on one axis. If a question above cannot be answered for a number, we say so rather than fill the gap — and we suggest you hold any table you rely on to the same standard. For a decision that matters, the most reliable benchmark is a small one built from your own tasks.