Evaluation
Benchmarks
- Benchmarks
- 14
- Results on file
- 35
- Last checked
How to read this catalogue
Provenance
Every number carries a label saying who produced it. Of the 35 results on file, 33 are vendor-reported, 2 are third-party. HARBAN has not yet published internal evaluations.
- Vendor-reportedPublished by the organisation that built the model. Not independently reproduced.
- Third-partyMeasured and published by an independent evaluator.
- HARBAN internalRun by HARBAN AI LABS with published settings and logs.
Health
Benchmarks age. Each one carries a dated health label — active, saturated, flawed or retired — backed by a public source, so you know how much weight a score can bear before you read it.
Comparability
Results are grouped by benchmark and variant — version, subset, tool setting — and are only charted against results from the same group. Vendor-reported numbers inside a group still use different settings.
Catalogue
All benchmarks (14)
| Benchmark | Category | Health | Year | Results on file | Metric |
|---|---|---|---|---|---|
| MMLUMeasuring Massive Multitask Language Understanding (MMLU) | Knowledge | Flawed | 2020 | None yet | Accuracy (%) |
| MMLU-Pro | Knowledge | Saturated | 2024 | 2 | Accuracy (%) |
| ARC-AGI-1ARC-AGI-1 (Abstraction and Reasoning Corpus) | Reasoning | Saturated | 2019 | None yet | Task accuracy (%) on the semi-private or private set (pass@2), reported with cost per task |
| ARC-AGI-2 | Reasoning | Active | 2025 | None yet | pass@2 accuracy (%) with cost per task |
| GPQA DiamondGPQA: A Graduate-Level Google-Proof Q&A Benchmark (Diamond subset) | Reasoning | Active | 2023 | 8 | Accuracy (%) |
| HLEHumanity's Last Exam (HLE) | Reasoning | Active | 2025 | 8 | Accuracy (%) (calibration error also reported) |
| AIMEAmerican Invitational Mathematics Examination (AIME), as used in LLM evals | Mathematics | Saturated | Not verifiedReference unconfirmed | 3 | Accuracy (%), usually mean over k samples per problem |
| MATHMATH (Measuring Mathematical Problem Solving With the MATH Dataset) | Mathematics | Flawed | 2021 | None yet | Exact-match accuracy (%) of final answer |
| HumanEval | Coding | Flawed | 2021 | None yet | pass@k (usually pass@1) |
| LiveCodeBench | Coding | Saturated | 2024 | 3 | pass@1 (%) within a stated problem date window |
| SWE-bench | Coding | Flawed | 2023 | None yet | % resolved |
| SWE-bench Verified | Coding | Flawed | 2024 | 4 | % resolved (of 500) |
| Terminal-Bench | Agents & tool use | Active | 2025 | 7 | Task resolution rate (%) with 95% CI, per benchmark version |
| τ-benchτ-bench (Tool-Agent-User Interaction benchmark) | Agents & tool use | Flawed | 2024 | None yet | pass^k (pass^1 = average task success; higher k = reliability across k trials) |
Health review
A health label is a dated judgement, not a permanent verdict. Each benchmark page gives the note behind its label, the date it was reviewed and the public source it rests on.
- Active
- No documented saturation, major flaw or retirement found as of the review date.
- 4 benchmarks
- Saturated
- Top models cluster near the ceiling; small differences are noise.
- 4 benchmarks
- Flawed
- Documented problems — label errors, weak tests or contamination — materially affect scores.
- 6 benchmarks
- Retired
- No longer maintained or superseded by a successor.
- 0 benchmarks