Benchmark · Knowledge
MMLU-Pro
Definition
- Short name
- MMLU-Pro
- Category
- Knowledge
- Introduced
- 2024
- Metric
- Accuracy (%)
- Direction
- Higher is better
- Reference check
- Confirmed against the paper or official site
Health review
As of
Top models cluster near the ceiling; small differences are noise.
Top models cluster: the three highest scores on Artificial Analysis’s MMLU-Pro leaderboard are 89.5–89.8%, and Artificial Analysis now lists it as a legacy evaluation.
Source: artificialanalysis.ai — source for this health review (external site)
Caveats
- Artificial Analysis removed MMLU-Pro from its Intelligence Index in v4.0 (January 2026), without stating a reason, and has retired it from active reporting.Source: artificialanalysis.ai — source for caveat 1 on MMLU-Pro (external site)
- Dataset revisions matter: the maintainers corrected 15 answers in the medical domain in April 2025.Source: huggingface.co — source for caveat 2 on MMLU-Pro (external site)
Results on file
2 results in 1 comparability group (2 vendor-reported). Results are only charted against others in the same group — the same benchmark and the same variant as named by the source. Last retrieved .
MMLU-Pro · No variant stated
2 results
The sources did not name a version or subset for these results.
Not directly comparable
These are vendor-reported numbers from 2 different organisations. Each source chose its own settings — reasoning effort, tools, harness or scaffold, number of attempts, grader — so a longer bar is a claim made under different conditions, not evidence that one model is better. None of these results has been reproduced by HARBAN.
| Model | Model detail | Score | Settings (as stated) | Source | Evidence |
|---|---|---|---|---|---|
| Gemma 4 31B | Not stated by source | 85.2%No CI reported | instruction-tuned model; no other conditions stated | Vendor-reportedGoogle DeepMindhuggingface.co — source for Gemma 4 31B on MMLU-Pro, No variant stated (external site) | Show excerpt for Gemma 4 31B on MMLU-Pro, No variant stated“| MMLU Pro | 85.2% |” Retrieved |
| Llama 4 Maverick | Not stated by source | 80.5%No CI reported | 0-shot; instruction-tuned model | Show excerpt for Llama 4 Maverick on MMLU-Pro, No variant stated“MMLU Pro 0 macro_avg/acc 68.9 73.4 74.3 80.5” Retrieved |
- Vendor-reported:
- Published by the organisation that built the model. Not independently reproduced.