Benchmark · Coding
HumanEval
Functional correctness of Python functions generated from docstrings, checked by unit tests.
Definition
- Short name
- HumanEval
- Category
- Coding
- Introduced
- 2021
- Metric
- pass@k (usually pass@1)
- Direction
- Higher is better
- Reference check
- Confirmed against the paper or official site
Health review
As of
Flawed
Documented problems — label errors, weak tests or contamination — materially affect scores.
Adding ~80× more tests (HumanEval+) lowered pass@k substantially, showing the original tests are weak.
Source: arxiv.org — source for this health review (external site)
Caveats
- Weak tests: adding roughly 80× more tests (HumanEval+) reduced pass@k by up to 19.3–28.9%.Source: arxiv.org — source for caveat 1 on HumanEval (external site)
Results on file
No results on file yet
No result for HumanEval is on file yet. We list a result only when we can trace it to a published source, and record the settings exactly as that source states them. HARBAN has not yet published internal evaluations of this benchmark.