Benchmark · Reasoning
GPQA: A Graduate-Level Google-Proof Q&A Benchmark (Diamond subset)
Definition
- Short name
- GPQA Diamond
- Category
- Reasoning
- Introduced
- 2023
- Metric
- Accuracy (%)
- Direction
- Higher is better
- Official site
- github.com — official site of GPQA: A Graduate-Level Google-Proof Q&A Benchmark (Diamond subset) (external site)
- Reference check
- Confirmed against the paper or official site
Health review
As of
No documented saturation, major flaw or retirement found as of the review date.
An Epoch AI post (Greg Burnham, May 2025) estimated roughly 8% of questions may be invalid and judged that saturation had not yet been reached. Artificial Analysis later removed GPQA Diamond from its Intelligence Index (v4.2) without stating a reason.
Source: epoch.ai — source for this health review (external site)
Caveats
- Small set: with 198 questions, error bars are wide. An Epoch AI post by Greg Burnham estimated about 8% of questions may be invalid and reported top models clustering around 83% (May 2025).Source: epoch.ai — source for caveat 1 on GPQA: A Graduate-Level Google-Proof Q&A Benchmark (Diamond subset) (external site)
- Artificial Analysis removed GPQA Diamond from its Intelligence Index in v4.2.Source: artificialanalysis.ai — source for caveat 2 on GPQA: A Graduate-Level Google-Proof Q&A Benchmark (Diamond subset) (external site)
Results on file
8 results in 1 comparability group (7 vendor-reported, 1 third-party). Results are only charted against others in the same group — the same benchmark and the same variant as named by the source. Last retrieved .
GPQA Diamond · No variant stated
8 results
The sources did not name a version or subset for these results.
Not directly comparable
These numbers mix provenance (7 vendor-reported, 1 third-party) from 6 different evaluators. Each source chose its own settings — reasoning effort, tools, harness or scaffold, number of attempts, grader — so a longer bar is a claim made under different conditions, not evidence that one model is better. None of these results has been reproduced by HARBAN.
| Model | Model detail | Score | Settings (as stated) | Source | Evidence |
|---|---|---|---|---|---|
| Gemini 3.1 Pro | Thinking (High) | 94.3%No CI reported | No tools; Thinking (High); all Gemini scores pass@1 unless noted (methodology PDF) | Vendor-reportedGoogle DeepMinddeepmind.google — source for Gemini 3.1 Pro (Thinking (High)) on GPQA Diamond, No variant stated (external site) | Show excerpt for Gemini 3.1 Pro (Thinking (High)) on GPQA Diamond, No variant stated“GPQA Diamond Scientific knowledge No tools 94.3%” Retrieved |
| Qwen3.8-Max | Not stated by source | 92.6%No CI reported | conditions not stated; column 'Qwen3.8-Max' on the Qwen3.8-2.4T-A95B card | Vendor-reportedAlibaba Cloud (Qwen)huggingface.co — source for Qwen3.8-Max on GPQA Diamond, No variant stated (external site) | Show excerpt for Qwen3.8-Max on GPQA Diamond, No variant stated“GPQA Diamond | 92.0 | 92.6 | 94.1 | 92.4 | 92.6 (last col = Qwen3.8-Max)” Retrieved |
| DeepSeek-V4.1-Flash | max reasoning effort | 90.9%No CI reported | Pass@1; max reasoning effort (reasoning_effort=100), temperature=1.0, top_p=0.95; last column = DS-V4.1-Flash | Show excerpt for DeepSeek-V4.1-Flash (max reasoning effort) on GPQA Diamond, No variant stated“GPQA Diamond (Pass@1) | 93.4 | **94.1** | 92.9 | 88.1 | 92.4 | 89.9 | 90.9” Retrieved | |
| Qwen3.8-27B | Not stated by source | 89.2%No CI reported | conditions not stated | Vendor-reportedAlibaba Cloud (Qwen)huggingface.co — source for Qwen3.8-27B on GPQA Diamond, No variant stated (external site) | Show excerpt for Qwen3.8-27B on GPQA Diamond, No variant stated“GPQA Diamond | 89.2 | 87.8 | 90.3 | 83.5 | 91.3 (first col = Qwen3.8-27B)” Retrieved |
| Gemma 4 31B | Not stated by source | 84.3%No CI reported | instruction-tuned model; no other conditions stated | Vendor-reportedGoogle DeepMindhuggingface.co — source for Gemma 4 31B on GPQA Diamond, No variant stated (external site) | Show excerpt for Gemma 4 31B on GPQA Diamond, No variant stated“| GPQA Diamond | 84.3% |” Retrieved |
| Muse Glimmer | High Reasoning | 83.5%No CI reported | Full 198-question set; high reasoning; temperature 1.0, top_p 0.95, top_k 64. The model card marks the row “(AA)”; Meta’s linked methodology report states GPQA Diamond results are sourced from Artificial Analysis. | Third-partyArtificial Analysis (republished by Meta)Muse Glimmer model card (Meta, Hugging Face) — source for Muse Glimmer (High Reasoning) on GPQA Diamond, No variant stated (external site) | Show excerpt for Muse Glimmer (High Reasoning) on GPQA Diamond, No variant stated“| GPQA Diamond (AA) | 83.5 | **85.7** | 84.2 |” Retrieved |
| gpt-oss-120b | reasoning level high | 80.1%No CI reported | no tools; reasoning level high (Table 3; columns low/medium/high) | Show excerpt for gpt-oss-120b (reasoning level high) on GPQA Diamond, No variant stated“GPQA Diamond (no tools) 67.1 73.1 80.1” Retrieved | |
| Llama 4 Maverick | Not stated by source | 69.8%No CI reported | 0-shot; instruction-tuned model | Show excerpt for Llama 4 Maverick on GPQA Diamond, No variant stated“GPQA Diamond 0 accuracy 50.5 49.0 57.2 69.8” Retrieved |
- Vendor-reported:
- Published by the organisation that built the model. Not independently reproduced.
- Third-party:
- Measured and published by an independent evaluator.