Benchmark · Knowledge
Measuring Massive Multitask Language Understanding (MMLU)
Broad academic and professional knowledge across 57 subjects via four-option multiple-choice questions.
Definition
- Short name
- MMLU
- Category
- Knowledge
- Introduced
- 2020
- Metric
- Accuracy (%)
- Direction
- Higher is better
- Paper
- arxiv.org — paper for Measuring Massive Multitask Language Understanding (MMLU) (external site)
- Official site
- github.com — official site of Measuring Massive Multitask Language Understanding (MMLU) (external site)
- Reference check
- Confirmed against the paper or official site
Health review
As of
Flawed
Documented problems — label errors, weak tests or contamination — materially affect scores.
An audit estimated that 6.49% of MMLU questions contain errors, concentrated in some subjects.
Source: arxiv.org — source for this health review (external site)
Caveats
- Label errors: an audit estimated that 6.49% of MMLU questions contain errors, with some subjects far worse (57% of analysed Virology questions).Source: arxiv.org — source for caveat 1 on Measuring Massive Multitask Language Understanding (MMLU) (external site)
Results on file
No results on file yet
No result for MMLU is on file yet. We list a result only when we can trace it to a published source, and record the settings exactly as that source states them. HARBAN has not yet published internal evaluations of this benchmark.