Benchmark · Mathematics
MATH (Measuring Mathematical Problem Solving With the MATH Dataset)
Competition-style math problem solving with exact final answers across 7 subjects and 5 difficulty levels (12,500 problems).
Definition
- Short name
- MATH
- Category
- Mathematics
- Introduced
- 2021
- Metric
- Exact-match accuracy (%) of final answer
- Direction
- Higher is better
- Official site
- github.com — official site of MATH (Measuring Mathematical Problem Solving With the MATH Dataset) (external site)
- Reference check
- Confirmed against the paper or official site
Health review
As of
Flawed
Documented problems — label errors, weak tests or contamination — materially affect scores.
Functionalised variants of MATH problems showed large accuracy gaps versus the static set under simple few-shot prompting, suggesting memorisation.
Source: arxiv.org — source for this health review (external site)
Caveats
- Static-set inflation: functionalised variants of MATH problems showed "reasoning gaps" of 58.35%–80.31% versus static scores, suggesting memorisation. The authors measured these gaps with a simple few-shot prompting strategy and expect them to be smaller with more sophisticated prompting.Source: arxiv.org — source for caveat 1 on MATH (Measuring Mathematical Problem Solving With the MATH Dataset) (external site)
Results on file
No results on file yet
No result for MATH is on file yet. We list a result only when we can trace it to a published source, and record the settings exactly as that source states them. HARBAN has not yet published internal evaluations of this benchmark.