Evaluation
Benchmarks
- Benchmarks
- 14
- Results on file
- 35
- Last checked
How to read this catalogue
Provenance
Every number carries a label saying who produced it. Of the 35 results on file, 33 are vendor-reported, 2 are third-party. HARBAN has not yet published internal evaluations.
- Vendor-reportedPublished by the organisation that built the model. Not independently reproduced.
- Third-partyMeasured and published by an independent evaluator.
- HARBAN internalRun by HARBAN AI LABS with published settings and logs.
Health
Benchmarks age. Each one carries a dated health label — active, saturated, flawed or retired — backed by a public source, so you know how much weight a score can bear before you read it.
Comparability
Results are grouped by benchmark and variant — version, subset, tool setting — and are only charted against results from the same group. Vendor-reported numbers inside a group still use different settings.
Catalogue
Benchmarks in the Reasoning category (4)
| Benchmark | Category | Health | Year | Results on file | Metric |
|---|---|---|---|---|---|
| ARC-AGI-1ARC-AGI-1 (Abstraction and Reasoning Corpus) | Reasoning | Saturated | 2019 | None yet | Task accuracy (%) on the semi-private or private set (pass@2), reported with cost per task |
| ARC-AGI-2 | Reasoning | Active | 2025 | None yet | pass@2 accuracy (%) with cost per task |
| GPQA DiamondGPQA: A Graduate-Level Google-Proof Q&A Benchmark (Diamond subset) | Reasoning | Active | 2023 | 8 | Accuracy (%) |
| HLEHumanity's Last Exam (HLE) | Reasoning | Active | 2025 | 8 | Accuracy (%) (calibration error also reported) |
Health review
A health label is a dated judgement, not a permanent verdict. Each benchmark page gives the note behind its label, the date it was reviewed and the public source it rests on.
- Active
- No documented saturation, major flaw or retirement found as of the review date.
- 4 benchmarks overall
- Saturated
- Top models cluster near the ceiling; small differences are noise.
- 4 benchmarks overall
- Flawed
- Documented problems — label errors, weak tests or contamination — materially affect scores.
- 6 benchmarks overall
- Retired
- No longer maintained or superseded by a successor.
- 0 benchmarks overall