How we source, label and compare benchmark numbers

Every result carries exactly one provenance label. The label says who ran the evaluation — not how good the number is.

A result is listed only if we can point to where it was published. For every external result we record:

• the source URL and the organisation that ran the evaluation;
• the benchmark variant exactly as the source names it — version, subset, tool setting or date window;
• the model identity as stated (snapshot, reasoning effort, mode);
• the settings as stated, or “conditions not stated” when the source gives none;
• a short verbatim excerpt from the source containing the number, so anyone can find it again;
• the date we retrieved it.

“Checked” means the number on this site matches the number on the source page on that date. It does not mean we reproduced the result, or that the evaluation was run well. Only a HARBAN internal result, published with its logs, means we ran it ourselves.

Two results can share an axis only when they come from the same benchmark and the same variant. That pair is the comparability group. A score on one version of a benchmark, or with tools versus without, is a different measurement — so it gets its own chart, or none.

Being in the same group is necessary but not sufficient: vendor-reported numbers inside one group are still produced under different settings, which is why every chart carries a warning.

Two vendors can report the same benchmark and still not be measuring the same thing. The usual reasons:

Settings

Reasoning effort, sampling temperature, token budgets, number of attempts and how answers are averaged. Effort labels such as “high” or “max” are defined by each vendor, so the same word need not mean the same setting at another vendor.

Harness or scaffold

Agentic benchmarks measure a model inside some software that gives it tools, memory and a loop. Change that software and the score can change, even with the same model.

Versions

Benchmarks are revised — tasks fixed, answer keys corrected, date windows moved — and a model name can cover more than one snapshot. A score belongs to one benchmark version and one model snapshot.

Grading and selection

Some benchmarks are graded by another model, and vendors choose which benchmarks and which configurations to publish.

A benchmark can stop being informative: top models bunch near the ceiling, errors are found in the answer key, or test data leaks into training sets. Each benchmark carries one of four labels, with a dated note and a public source for the judgement.

The model registry shows a field — release date, context window, modalities, licence — only when it appears on an official page from the model’s developer, and it links that page next to the value. We keep the vendor’s own wording (“1M tokens”) beside the normalised number, because the two do not always agree exactly.

Anything we could not confirm is shown as “Not verified”. We never fill a gap with a third-party figure or an estimate.

HARBAN has not yet published any internal evaluations. When we do, a result will only carry the “HARBAN internal” label if it is published with:

• the exact model identity and the date it was accessed;
• the full settings — prompt template, sampling parameters, reasoning effort, tools and token limits;
• the harness or scaffold and its version, and the benchmark version and any excluded items;
• the number of runs, with per-run scores;
• a confidence interval and how it was computed;
• complete run logs, linked from the result.

Our data schema enforces the last point: a result cannot be labelled internal without a link to its logs.

If a number, date or label is wrong or out of date, use “Report an issue” on the benchmark or model page and include a link to the source. How we review and log corrections is set out in our editorial and data standards.