Benchmark · Coding
SWE-bench Verified
Definition
- Short name
- SWE-bench Verified
- Category
- Coding
- Introduced
- 2024
- Metric
- % resolved (of 500)
- Direction
- Higher is better
- Paper
- No paper
- Reference check
- Confirmed against the paper or official site
Health review
As of
Documented problems — label errors, weak tests or contamination — materially affect scores.
Epoch AI’s benchmark review gives SWE-bench Verified a "Flawed" verdict (September 2026).
Source: epoch.ai — source for this health review (external site)
Caveats
- Epoch AI’s review verdict is "Flawed" (reviewed 3 Sep 2026). It summarises an OpenAI audit (Feb 2026) that examined 27.6% of tasks and found 59.4% of those had flawed test cases.Source: epoch.ai — source for caveat 1 on SWE-bench Verified (external site)
- Since 18 Nov 2025, the maintainers accept new Verified submissions only from academic teams and research institutions that publish their methods.Source: github.com — source for caveat 2 on SWE-bench Verified (external site)
Results on file
4 results in 1 comparability group (4 vendor-reported). Results are only charted against others in the same group — the same benchmark and the same variant as named by the source. Last retrieved .
SWE-bench Verified · No variant stated
4 results
The sources did not name a version or subset for these results.
Not directly comparable
These are vendor-reported numbers from 4 different organisations. Each source chose its own settings — reasoning effort, tools, harness or scaffold, number of attempts, grader — so a longer bar is a claim made under different conditions, not evidence that one model is better. None of these results has been reproduced by HARBAN.
| Model | Model detail | Score | Settings (as stated) | Source | Evidence |
|---|---|---|---|---|---|
| Gemini 3.1 Pro | Thinking (High) | 80.6%No CI reported | Single attempt; methodology PDF says averaged over 10 runs with a bash + file-edit + submit-tool scaffold, and that the score includes a +0.6-point adjustment after Google fixed 3 official-harness items (astropy-7606, sphinx-8595, sphinx-9711) the model passed internally | Vendor-reportedGoogle DeepMinddeepmind.google — source for Gemini 3.1 Pro (Thinking (High)) on SWE-bench Verified, No variant stated (external site) | Show excerpt for Gemini 3.1 Pro (Thinking (High)) on SWE-bench Verified, No variant stated“SWE-Bench Verified Agentic coding Single attempt 80.6%” Retrieved |
| Mistral Medium 3.5 | Not stated by source | 77.6%No CI reported | conditions not stated | Show excerpt for Mistral Medium 3.5 on SWE-bench Verified, No variant stated“It scores **91.4%** on τ³-Telecom and **77.6%** on SWE-Bench Verified.” Retrieved | |
| Muse Glimmer | High Reasoning | 76.0%No CI reported | High reasoning; scaffold with a bash tool and a file-operation tool; average pass rate over 500 tasks, averaged across 4 runs (Meta methodology report). | Show excerpt for Muse Glimmer (High Reasoning) on SWE-bench Verified, No variant stated“| SWE-Bench Verified | 76.0 | 66.6 | **77.2** |” Retrieved | |
| gpt-oss-120b | reasoning level high | 62.4%No CI reported | reasoning level high (Table 3) | Show excerpt for gpt-oss-120b (reasoning level high) on SWE-bench Verified, No variant stated“SWE-Bench Verified 47.9 52.6 62.4” Retrieved |
- Vendor-reported:
- Published by the organisation that built the model. Not independently reproduced.