Service · AI/LLM evaluation & benchmarking
Reliability & red-team evaluation
Being designed. Not yet offered — register interest via Work with us.
At a glance
Status
We are designing this service line and are not taking engagements for it yet. The approach and deliverables on this page describe how we intend to run it and may change before it opens. If its status changes, the new status will be shown here and on the services index.
- Who it’s for
- Teams deploying agents or LLM applications · AI vendors preparing a release
- Practice
- AI/LLM evaluation & benchmarking
- Topics
- Safety & reliabilityAgentsEvaluation
- Solutions
- AI labs & model vendors
Contents
The problem
LLM systems fail in ways unit tests do not cover: injected instructions in retrieved content, tools called with unsafe arguments, confident answers outside the system's competence.
Approach
How we intend to run this work, step by step.
Step 1: Threat model the system
Map inputs, tools, data access and trust boundaries; agree what counts as a failure and what is out of scope.
Step 2: Design attack and stress suites
Targeted test cases for injection, jailbreaks, tool misuse, leakage and degraded inputs, derived from the threat model.
Step 3: Measure and reproduce
Report failure rates with exact reproduction steps, not anecdotes.
Step 4: Recommend mitigations
Prioritised fixes — guardrails, permission scoping, output validation — and a re-test to confirm they work.
Deliverables
What an engagement is designed to produce. The final list is fixed in the written proposal.
- Threat model and agreed rules of engagement
- Reproducible adversarial test suite
- Findings report with severity, reproduction steps and mitigations
- Re-test results after fixes
What this is not
- Not yet offered. This page describes a planned service line, not a service you can book today.
- Testing happens only on systems you own or are authorised to test, under written rules of engagement.
- Red-team findings reduce risk; they cannot prove a system is safe.
- Not a fixed package. Scope, deliverables and acceptance criteria are agreed in writing for each engagement.
Related benchmarks
Public benchmarks relevant to this work, with our review of what each measures and how healthy it is. They are a reference point, not a substitute for evaluation on your own tasks.
Related research & insights
Paper breakdowns and guides on the topics this service draws on: Safety & reliability · Agents · Evaluation.
Analysis ·
How to read an AI benchmark table without fooling yourself
Eight questions to ask before you trust a row on a leaderboard: who ran it, which version, under what settings, whether the benchmark still separates models, whether the model has seen it, what the grader checks, how big the noise is, and which harness produced it.
Paper breakdown ·
ReAct: interleaving reasoning and actions in language models
A technical reading of the 2022 paper that had language models alternate between written reasoning and tool actions — how the thought–action–observation loop works, what it improved on question answering and interactive tasks, and where it failed.
Guide ·
A practical checklist for shipping a RAG system
What to have in place before a retrieval-augmented generation system meets real users: a retrieval evaluation set, structure-aware chunking, hybrid search, verifiable citations, permission-aware retrieval, a freshness plan and production monitoring.
Explainer ·
What counts as an AI agent? A working definition for engineers
A definition you can check against code: a model, a set of tools, a loop, state carried between steps and an explicit stop condition — plus how to tell when a fixed workflow is the better engineering choice.
Questions
Would you test production systems?
Only with explicit written authorisation and agreed limits, once this service opens. A staging environment would be preferred.
Can we engage you for this now?
Not yet. This service line is planned: we are designing it and are not taking engagements for it. You can still register interest and describe what you need — it creates no obligation on either side.
Do you publish prices or timelines?
No. Scope, deliverables, acceptance criteria and any timing are agreed in a written proposal before work starts. We do not publish delivery service levels we have not committed to.
Can we share confidential material in an enquiry?
Please don’t. Describe the problem in general terms; no confidentiality obligation arises until an NDA is signed.