Benchmark · Agents & tool use
Terminal-Bench
Definition
- Short name
- Terminal-Bench
- Category
- Agents & tool use
- Introduced
- 2025
- Metric
- Task resolution rate (%) with 95% CI, per benchmark version
- Direction
- Higher is better
- Reference check
- Confirmed against the paper or official site
Health review
As of
No documented saturation, major flaw or retirement found as of the review date.
Actively maintained: version 4.0 (dated 28 Aug 2026 on the Terminal-Bench blog) removed eight tasks — two of them for saturation — and fixed 19 others. Compare scores only within one version.
Source: tbench.ai — source for this health review (external site)
Caveats
- Version-sensitive: 2.1 fixed issues in 28 of the 89 tasks in 2.0, and 4.0 introduced further breaking changes; scores are not comparable across versions.Source: tbench.ai — source for caveat 1 on Terminal-Bench (external site)
- Harness matters: the same model can score very differently depending on the agent harness used.Source: tbench.ai — source for caveat 2 on Terminal-Bench (external site)
- Version 1.0 included unstable tasks (for example download-youtube), which 2.0 addressed.Source: tbench.ai — source for caveat 3 on Terminal-Bench (external site)
Results on file
7 results in 3 comparability groups (6 vendor-reported, 1 third-party). Results are only charted against others in the same group — the same benchmark and the same variant as named by the source. Last retrieved .
Terminal-Bench · 2.1
4 results
Not directly comparable
These numbers mix provenance (3 vendor-reported, 1 third-party) from 3 different evaluators. Each source chose its own settings — reasoning effort, tools, harness or scaffold, number of attempts, grader — so a longer bar is a claim made under different conditions, not evidence that one model is better. None of these results has been reproduced by HARBAN.
| Model | Model detail | Score | Settings (as stated) | Source | Evidence |
|---|---|---|---|---|---|
| DeepSeek-V4.1-Flash | max reasoning effort | 90.6%No CI reported | Pass@1; max reasoning effort; Minimal mode of DeepSeek Harness, 1M-token context window | Show excerpt for DeepSeek-V4.1-Flash (max reasoning effort) on Terminal-Bench, 2.1“Terminal-Bench 2.1 (Pass@1) | 89.1 | 88.8 | 88.3 | 88.2 | 87.9 | 82.7 | **90.6**” Retrieved | |
| Qwen3.8-Max | Not stated by source | 86.6%No CI reported | Evaluated with Claude Code (avg@10), 5-hour timeout, max_tokens=131,072 (card footnote 2) | Vendor-reportedAlibaba Cloud (Qwen)huggingface.co — source for Qwen3.8-Max on Terminal-Bench, 2.1 (external site) | Show excerpt for Qwen3.8-Max on Terminal-Bench, 2.1“Terminal Bench 2.1 | 84.6 | 84.6 | 88.8 | 74.5 | 86.6 (last col = Qwen3.8-Max)” Retrieved |
| Qwen3.8-27B | Not stated by source | 73.0%No CI reported | Terminus harness (row label 'Terminal Bench 2.1 (Terminus)') | Vendor-reportedAlibaba Cloud (Qwen)huggingface.co — source for Qwen3.8-27B on Terminal-Bench, 2.1 (external site) | Show excerpt for Qwen3.8-27B on Terminal-Bench, 2.1“Terminal Bench 2.1 (Terminus) | 73.0 | 63.4 | 64.0 | 51.7 | 78.2” Retrieved |
| Muse Glimmer | High Reasoning | 51.7%No CI reported | with terminus2; High Reasoning. Meta’s linked methodology report states Terminal-Bench 2.1 results are sourced from Artificial Analysis (Terminus 2 harness in an E2B sandbox; 89 tasks). | Third-partyArtificial Analysis (republished by Meta)Muse Glimmer model card (Meta, Hugging Face) — source for Muse Glimmer (High Reasoning) on Terminal-Bench, 2.1 (external site) | Show excerpt for Muse Glimmer (High Reasoning) on Terminal-Bench, 2.1“| TerminalBench 2.1 (with terminus2) | 51.7 | 43.4 | **60.7** |” Retrieved |
- Vendor-reported:
- Published by the organisation that built the model. Not independently reproduced.
- Third-party:
- Measured and published by an independent evaluator.
Terminal-Bench · 4.0
2 results
Not directly comparable
These are vendor-reported numbers from 2 different organisations. Each source chose its own settings — reasoning effort, tools, harness or scaffold, number of attempts, grader — so a longer bar is a claim made under different conditions, not evidence that one model is better. None of these results has been reproduced by HARBAN.
| Model | Model detail | Score | Settings (as stated) | Source | Evidence |
|---|---|---|---|---|---|
| Claude Opus 5.5 | xhigh effort | 66.4%No CI reported | xhigh effort (per page footnote); stated standard error ±2.6 pts. Anthropic evaluated Opus 5.5 with production safeguards enabled; when they intervened, biology and frontier LLM development tasks were completed by Claude Opus 5 and cybersecurity tasks by Claude Opus 4.8 (page note). | Vendor-reportedAnthropicanthropic.com — source for Claude Opus 5.5 (xhigh effort) on Terminal-Bench, 4.0 (external site) | Show excerpt for Claude Opus 5.5 (xhigh effort) on Terminal-Bench, 4.0“Agentic coding Terminal-Bench 4.0¹ 66.4%” Retrieved |
| Grok 4.7 | xHigh | 38.0%No CI reported | xhigh reasoning; Grok Build harness (model card §2.3). xAI’s announcement page reports 37.6% for the same model — the two official sources disagree. | Show excerpt for Grok 4.7 (xHigh) on Terminal-Bench, 4.0“Grok 4.7 scores 38.0% at xhigh, evaluated with the Grok Build harness.” Retrieved |
- Vendor-reported:
- Published by the organisation that built the model. Not independently reproduced.
Terminal-Bench · 2.0
1 result
Not directly comparable
These are vendor-reported numbers. Each source chose its own settings — reasoning effort, tools, harness or scaffold, number of attempts, grader — so a longer bar is a claim made under different conditions, not evidence that one model is better. None of these results has been reproduced by HARBAN.
Only one result is on file for this variant, so there is nothing to chart it against.
| Model | Model detail | Score | Settings (as stated) | Source | Evidence |
|---|---|---|---|---|---|
| Gemini 3.1 Pro | Thinking (High) | 68.5%No CI reported | Terminus-2 harness; self-computed | Vendor-reportedGoogle DeepMinddeepmind.google — source for Gemini 3.1 Pro (Thinking (High)) on Terminal-Bench, 2.0 (external site) | Show excerpt for Gemini 3.1 Pro (Thinking (High)) on Terminal-Bench, 2.0“Terminal-Bench 2.0 Agentic terminal coding Terminus-2 harness 68.5%” Retrieved |
- Vendor-reported:
- Published by the organisation that built the model. Not independently reproduced.