Benchmark · Coding
SWE-bench
Whether a system can resolve real GitHub issues in 12 Python repositories by producing a patch that passes the repository's tests (2,294 tasks).
Definition
- Short name
- SWE-bench
- Category
- Coding
- Introduced
- 2023
- Metric
- % resolved
- Direction
- Higher is better
- Reference check
- Confirmed against the paper or official site
Health review
As of
Flawed
Documented problems — label errors, weak tests or contamination — materially affect scores.
SWE-Bench+ found solution leakage in issue text and weak tests behind many "successful" patches.
Source: arxiv.org — source for this health review (external site)
Caveats
- SWE-Bench+ found 32.67% of successful patches had solutions leaked in the issue text and 31.08% passed due to weak tests; >94% of issues predate model cutoffs.Source: arxiv.org — source for caveat 1 on SWE-bench (external site)
Results on file
No results on file yet
No result for SWE-bench is on file yet. We list a result only when we can trace it to a published source, and record the settings exactly as that source states them. HARBAN has not yet published internal evaluations of this benchmark.