Benchmark · Agents & tool use
τ-bench (Tool-Agent-User Interaction benchmark)
Whether a tool-using agent completes customer-service tasks (retail, airline) while following domain policy in conversation with an LLM-simulated user.
Definition
- Short name
- τ-bench
- Category
- Agents & tool use
- Introduced
- 2024
- Metric
- pass^k (pass^1 = average task success; higher k = reliability across k trials)
- Direction
- Higher is better
- Official site
- github.com — official site of τ-bench (Tool-Agent-User Interaction benchmark) (external site)
- Reference check
- Confirmed against the paper or official site
Health review
As of
Flawed
Documented problems — label errors, weak tests or contamination — materially affect scores.
An audit found τ-bench can count empty responses as successful; the maintainers’ README says the original tasks are outdated and points to the successor τ³-bench.
Source: arxiv.org — source for this health review (external site)
Caveats
- Grading flaw: an audit of agentic benchmarks found τ-bench counts empty responses as successful.Source: arxiv.org — source for caveat 1 on τ-bench (Tool-Agent-User Interaction benchmark) (external site)
- Superseded: the maintainers describe the original tasks as outdated and recommend τ³-bench.Source: github.com — source for caveat 2 on τ-bench (Tool-Agent-User Interaction benchmark) (external site)
Results on file
No results on file yet
No result for τ-bench is on file yet. We list a result only when we can trace it to a published source, and record the settings exactly as that source states them. HARBAN has not yet published internal evaluations of this benchmark.