Benchmark · Reasoning
Humanity's Last Exam (HLE)
Definition
- Short name
- HLE
- Category
- Reasoning
- Introduced
- 2025
- Metric
- Accuracy (%) (calibration error also reported)
- Direction
- Higher is better
- Reference check
- Confirmed against the paper or official site
Health review
As of
No documented saturation, major flaw or retirement found as of the review date.
No saturation or retirement notice found as of the review date; see the label-error caveat.
Source: lastexam.ai — source for this health review (external site)
Caveats
- Label errors: FutureHouse found 29 ± 3.7% of text-only chemistry and biology answers conflicted with peer-reviewed literature (July 2025). Question selection deliberately targeted questions frontier models could not answer.Source: futurehouse.org — source for caveat 1 on Humanity's Last Exam (HLE) (external site)
- Versions matter: the maintainers released HLE-Rolling (Oct 2025) and HLE-Diamond, a refined 1,000-question subset (22 Sep 2026). State which set a score uses.Source: lastexam.ai — source for caveat 2 on Humanity's Last Exam (HLE) (external site)
Results on file
8 results in 3 comparability groups (8 vendor-reported). Results are only charted against others in the same group — the same benchmark and the same variant as named by the source. Last retrieved .
HLE · No tools
3 results
Not directly comparable
These are vendor-reported numbers from 2 different organisations. Each source chose its own settings — reasoning effort, tools, harness or scaffold, number of attempts, grader — so a longer bar is a claim made under different conditions, not evidence that one model is better. None of these results has been reproduced by HARBAN.
| Model | Model detail | Score | Settings (as stated) | Source | Evidence |
|---|---|---|---|---|---|
| Gemini 3.1 Pro | Thinking (High) | 44.4%No CI reported | No tools; full set, text + MM; column 'Gemini 3.1 Pro Thinking (High)' | Vendor-reportedGoogle DeepMinddeepmind.google — source for Gemini 3.1 Pro (Thinking (High)) on HLE, No tools (external site) | Show excerpt for Gemini 3.1 Pro (Thinking (High)) on HLE, No tools“Humanity's Last Exam Academic reasoning (full set, text + MM) No tools 44.4%” Retrieved |
| Gemma 4 31B | Not stated by source | 19.5%No CI reported | no tools; instruction-tuned model | Vendor-reportedGoogle DeepMindhuggingface.co — source for Gemma 4 31B on HLE, No tools (external site) | Show excerpt for Gemma 4 31B on HLE, No tools“| HLE no tools | 19.5% |” Retrieved |
| gpt-oss-120b | reasoning level high | 14.9%No CI reported | no tools; reasoning level high (Table 3) | Show excerpt for gpt-oss-120b (reasoning level high) on HLE, No tools“HLE (no tools) 5.2 8.6 14.9” Retrieved |
- Vendor-reported:
- Published by the organisation that built the model. Not independently reproduced.
HLE · Tools not stated
3 results
Not directly comparable
These are vendor-reported numbers from 2 different organisations. Each source chose its own settings — reasoning effort, tools, harness or scaffold, number of attempts, grader — so a longer bar is a claim made under different conditions, not evidence that one model is better. None of these results has been reproduced by HARBAN.
| Model | Model detail | Score | Settings (as stated) | Source | Evidence |
|---|---|---|---|---|---|
| Qwen3.8-Max | Not stated by source | 43.6%No CI reported | conditions not stated (a separate 'HLE w/ tools' row gives 56.2) | Vendor-reportedAlibaba Cloud (Qwen)huggingface.co — source for Qwen3.8-Max on HLE, Tools not stated (external site) | Show excerpt for Qwen3.8-Max on HLE, Tools not stated“HLE | 45.7 | 53.3 | 47.2 | 41.4 | 43.6 (last col = Qwen3.8-Max)” Retrieved |
| DeepSeek-V4.1-Flash | max reasoning effort | 36.8%No CI reported | Pass@1; max reasoning effort; tool setting not stated on this row (a separate 'HLE w/ tools' row gives 63.9); 39.1 on the text-only subset | Show excerpt for DeepSeek-V4.1-Flash (max reasoning effort) on HLE, Tools not stated“HLE (Pass@1) | **56.3** | 44.5 | 43.5 | 42.0† | 42.7† | 37.8† | 36.8 (39.1†)” Retrieved | |
| Qwen3.8-27B | Not stated by source | 30.8%No CI reported | HLE: Judged by GPT-4o (card footnote) | Vendor-reportedAlibaba Cloud (Qwen)huggingface.co — source for Qwen3.8-27B on HLE, Tools not stated (external site) | Show excerpt for Qwen3.8-27B on HLE, Tools not stated“HLE | 30.8 | 24.0 | 34.7 | 22.0 | 40.0 (first col = Qwen3.8-27B)” Retrieved |
- Vendor-reported:
- Published by the organisation that built the model. Not independently reproduced.
HLE · With tools
2 results
Not directly comparable
These are vendor-reported numbers from 2 different organisations. Each source chose its own settings — reasoning effort, tools, harness or scaffold, number of attempts, grader — so a longer bar is a claim made under different conditions, not evidence that one model is better. None of these results has been reproduced by HARBAN.
| Model | Model detail | Score | Settings (as stated) | Source | Evidence |
|---|---|---|---|---|---|
| Claude Opus 5.5 | adaptive thinking at max effort | 67.7%No CI reported | with tools; adaptive thinking at max effort (page default for all Opus 5.5 results unless otherwise noted). Anthropic evaluated Opus 5.5 with production safeguards enabled; when they intervened, biology and frontier LLM development tasks were completed by Claude Opus 5 and cybersecurity tasks by Claude Opus 4.8 (page note). | Show excerpt for Claude Opus 5.5 (adaptive thinking at max effort) on HLE, With tools“Humanity's Last Exam 67.7% with tools” Retrieved | |
| Gemini 3.1 Pro | Thinking (High) | 51.4%No CI reported | Search (blocklist) + Code; Thinking (High) | Vendor-reportedGoogle DeepMinddeepmind.google — source for Gemini 3.1 Pro (Thinking (High)) on HLE, With tools (external site) | Show excerpt for Gemini 3.1 Pro (Thinking (High)) on HLE, With tools“Search (blocklist) + Code 51.4%” Retrieved |
- Vendor-reported:
- Published by the organisation that built the model. Not independently reproduced.