AI research · evaluation · engineering
Research.
Benchmark.
Build.
HARBAN AI LABS is an AI research, evaluation and engineering company. We publish paper breakdowns, review benchmarks and keep a sourced model registry — every number sourced, every claim dated — and we are building evaluation and AI engineering services.
01 — Research
We read the papers closely
Technical breakdowns of influential AI research: what a paper claims, how the method works, and what it does not show.
02 — Evaluation
Evaluation with provenance
Benchmark reviews and a model registry of officially sourced facts. Every result is labelled by who produced it and how — so far, model vendors and independent evaluators; we have not published results of our own yet.
03 — Engineering
Engineering, evaluation first
Reference architectures and guides for agents, retrieval systems and LLM applications, designed around measurable behaviour. Our engineering service lines are planned, not yet offered.
Evaluation
Which benchmarks still mean something?
GPQA Diamond · Reasoning
ActiveAn Epoch AI post (Greg Burnham, May 2025) estimated roughly 8% of questions may be invalid and judged that saturation had not yet been reached. Artificial Analysis later removed GPQA Diamond from its Intelligence Index (v4.2) without stating a reason.
HLE · Reasoning
ActiveNo saturation or retirement notice found as of the review date; see the label-error caveat.
Terminal-Bench · Agents & tool use
ActiveActively maintained: version 4.0 (dated 28 Aug 2026 on the Terminal-Bench blog) removed eight tasks — two of them for saturation — and fixed 19 others. Compare scores only within one version.
SWE-bench Verified · Coding
FlawedEpoch AI’s benchmark review gives SWE-bench Verified a "Flawed" verdict (September 2026).
MMLU · Knowledge
FlawedAn audit estimated that 6.49% of MMLU questions contain errors, concentrated in some subjects.
MMLU-Pro · Knowledge
SaturatedTop models cluster: the three highest scores on Artificial Analysis’s MMLU-Pro leaderboard are 89.5–89.8%, and Artificial Analysis now lists it as a legacy evaluation.
Showing 6 of 14 reviewed benchmarks. See all reviews
Research
Latest paper breakdowns
2017 · Architectures, Large language models
Attention Is All You Need: the architecture behind modern language models
A technical reading of the 2017 paper that replaced recurrence with self-attention — what the Transformer does, why it trains faster, and what the paper did and did not show.
2022 · Prompting, Reasoning, Large language models
Chain-of-Thought Prompting: eliciting step-by-step reasoning with a few examples
A technical reading of the 2022 paper showing that few-shot examples with written-out reasoning steps improve large language models on arithmetic, commonsense and symbolic tasks — and why the effect appeared only at large scale.
2022 · Scaling, Large language models
Chinchilla: how to split a compute budget between model size and training data
A technical reading of the 2022 paper that concluded large language models were undertrained — the three ways it estimated compute-optimal scaling, the Chinchilla-versus-Gopher test, and why later work revisits the recipe.
For teams
Services we are building
- Planned
LLM & model evaluation
Task-specific evaluation suites and benchmark analysis that answer a concrete decision — which model to ship, whether a release regressed, where a system fails.
- Planned
Reliability & red-team evaluation
Structured adversarial testing of LLM applications and agents — prompt injection, unsafe tool use, data leakage and failure under stress.
- Planned
AI agent engineering
Design and build of tool-using agents with explicit control flow, scoped permissions, evaluation and observability from day one.
- Planned
LLM application development
Production LLM features and applications — from API integration and prompt design to structured outputs, testing and deployment.
- Planned
RAG & knowledge systems
Retrieval-augmented systems over your documents and data, measured on retrieval quality and answer faithfulness — not demos.
- Planned
AI automation & data workflows
Automations that combine deterministic pipelines with model steps — document processing, classification, extraction and routing.
- Planned
Prototypes & proofs of concept
Time-boxed builds that answer one question — is this feasible, and how well does it work on your data?
- Planned
Technical research & advisory
Literature reviews, model and vendor assessments, experiment design and technical due diligence for AI initiatives.
Building something and want an honest view of whether we can help? Describe the problem — you get a reference number as soon as you submit the form.
Start an enquiryLearn
Insights for practitioners
Analysis
How to read an AI benchmark table without fooling yourself
Eight questions to ask before you trust a row on a leaderboard: who ran it, which version, under what settings, whether the benchmark still separates models, whether the model has seen it, what the grader checks, how big the noise is, and which harness produced it.
Guide
A practical checklist for shipping a RAG system
What to have in place before a retrieval-augmented generation system meets real users: a retrieval evaluation set, structure-aware chunking, hybrid search, verifiable citations, permission-aware retrieval, a freshness plan and production monitoring.
Explainer
What counts as an AI agent? A working definition for engineers
A definition you can check against code: a model, a set of tools, a loop, state carried between steps and an explicit stop condition — plus how to tell when a fixed workflow is the better engineering choice.
How we work
Standards instead of slogans
01
Every number has a source
Model facts, benchmark results and tool details link to the primary page they came from, with the date we checked.
02
“Not verified” beats a guess
When a fact can’t be confirmed on an official source, we show that plainly instead of filling the gap.
03
Provenance is labelled
Vendor-reported, third-party and internal results are never mixed silently or charted as if comparable.
04
Corrections are public
Paper breakdowns, articles, projects and every model, benchmark, tool and service page have a “Report an issue” form. Breakdowns and articles log confirmed corrections with dates.
Newsletter
New research and data, by email
Paper breakdowns, benchmark reviews and registry updates. No fixed schedule and no filler — the first issue has not gone out yet.