Service · AI/LLM evaluation & benchmarking
LLM & model evaluation
Being designed. Not yet offered — register interest via Work with us.
At a glance
Status
We are designing this service line and are not taking engagements for it yet. The approach and deliverables on this page describe how we intend to run it and may change before it opens. If its status changes, the new status will be shown here and on the services index.
- Who it’s for
- AI labs & model vendors · Product teams shipping LLM features
- Practice
- AI/LLM evaluation & benchmarking
Contents
The problem
Public leaderboards are measured under settings that rarely match your task, data or risk tolerance. Teams choose and ship models on numbers that were never designed to answer their question.
Approach
How we intend to run this work, step by step.
Step 1: Define the decision
Agree what the evaluation must decide — model selection, a release gate, a regression check — and the threshold that changes the decision.
Step 2: Review the public benchmarks
Check which public benchmarks, if any, measure what the decision needs — using their documented caveats and health labels — and design a task-specific benchmark where none does.
Step 3: Build a representative test set
Assemble cases from real usage or carefully reviewed synthetic data, with a held-out split that is never used for prompt tuning.
Step 4: Choose metrics and graders
Prefer exact checks (unit tests, exact match) and validate any LLM-as-judge grader against human labels before trusting it.
Step 5: Run with full provenance
Fixed, published settings; repeated runs with uncertainty estimates; complete logs so every number can be traced.
Deliverables
What an engagement is designed to produce. The final list is fixed in the written proposal.
- Evaluation plan: decision, metrics, acceptance thresholds
- Benchmark review: which public results bear on the decision, and which do not
- Versioned test set and runnable evaluation harness
- Results report with settings, uncertainty and failure analysis
- Regression pipeline that can be re-run on new model versions
What this is not
- Not yet offered. This page describes a planned service line, not a service you can book today.
- Results describe the tested tasks and settings only; they are not a certification.
- Benchmark scores from vendors are reported as vendor-reported and never mixed with measured results.
- Not a fixed package. Scope, deliverables and acceptance criteria are agreed in writing for each engagement.
Related benchmarks
Public benchmarks relevant to this work, with our review of what each measures and how healthy it is. They are a reference point, not a substitute for evaluation on your own tasks.
- GPQA DiamondReasoningGraduate-level biology, physics and chemistry multiple-choice questions written to resist web lookup; Diamond is the 198-question highest-agreement subset.Active
- SWE-bench VerifiedCodingA 500-task human-screened subset of SWE-bench (built with OpenAI) where issues are judged clear and tests valid.Flawed
- Terminal-BenchAgents & tool useWhether an agent can complete realistic, test-verified tasks in a sandboxed command-line environment.Active
Related research & insights
Paper breakdowns and guides on the topics this service draws on: Evaluation · Large language models.
Analysis ·
How to read an AI benchmark table without fooling yourself
Eight questions to ask before you trust a row on a leaderboard: who ran it, which version, under what settings, whether the benchmark still separates models, whether the model has seen it, what the grader checks, how big the noise is, and which harness produced it.
Paper breakdown ·
Attention Is All You Need: the architecture behind modern language models
A technical reading of the 2017 paper that replaced recurrence with self-attention — what the Transformer does, why it trains faster, and what the paper did and did not show.
Paper breakdown ·
Chain-of-Thought Prompting: eliciting step-by-step reasoning with a few examples
A technical reading of the 2022 paper showing that few-shot examples with written-out reasoning steps improve large language models on arithmetic, commonsense and symbolic tasks — and why the effect appeared only at large scale.
Paper breakdown ·
Chinchilla: how to split a compute budget between model size and training data
A technical reading of the 2022 paper that concluded large language models were undertrained — the three ways it estimated compute-optimal scaling, the Chinchilla-versus-Gopher test, and why later work revisits the recipe.
Paper breakdown ·
LoRA: fine-tuning large models by training small low-rank updates
A technical reading of the 2021 paper that freezes pre-trained weights and learns low-rank update matrices instead — the maths, why it adds no inference latency, what the authors report against full fine-tuning, and what remains open.
Paper breakdown ·
Retrieval-Augmented Generation: pairing a language model with a searchable memory
A technical reading of the 2020 paper that introduced RAG models — how a dense retriever and a sequence-to-sequence generator are trained together, the difference between RAG-Sequence and RAG-Token, and what the results do and do not show.
Questions
Can you just run the popular public benchmarks?
Once this service opens, yes — but many of the public benchmarks we review are saturated or have documented flaws (see our benchmark pages). They are a sanity check, not an answer to a product decision.
Would you use LLM-as-judge?
Only after measuring its agreement with human labels on your data. Where a deterministic check exists, we would prefer it.
Can we engage you for this now?
Not yet. This service line is planned: we are designing it and are not taking engagements for it. You can still register interest and describe what you need — it creates no obligation on either side.
Do you publish prices or timelines?
No. Scope, deliverables, acceptance criteria and any timing are agreed in a written proposal before work starts. We do not publish delivery service levels we have not committed to.
Can we share confidential material in an enquiry?
Please don’t. Describe the problem in general terms; no confidentiality obligation arises until an NDA is signed.