AI engineering
AI engineering, evaluation first
Status. Our engineering service lines are planned: we are designing them and are not yet taking engagements. Each one shows its status on its service page.
Principles
Five principles we design to
Evaluation first
Agree how success will be measured before building — a representative test set, a held-out split and acceptance thresholds — then check every prompt, model or code change against them.
Least privilege for agents
Agents get the smallest tool surface that does the job: typed tool inputs, scoped credentials, explicit stop conditions, and a person’s approval before anything irreversible.
Observability
Every model call and tool call is traced with inputs, outputs, cost and latency, so a failure can be reproduced, explained and added to the test set.
Simplest design that works
Deterministic code where it is enough, a fixed workflow before an agent, one model call before a chain. When a simpler design meets the requirement, we recommend it.
Provenance
Model identifiers, prompts, data versions and settings are recorded with every result, so any number we report can be traced back to how it was produced.
Reference architectures
Three systems, drawn minimally
The shapes we start from. Each diagram shows structure and control points, not a specific product or vendor; real designs add what the requirements demand and nothing more.
Reference architecture 1
Agent loop with explicit controls
An agent is a loop: the model proposes an action, the system decides whether it may run, and the result feeds the next step. The controls are enforced in code outside the model, so nothing the model writes can switch them off.
Dashed outline: a control point the agent cannot skip.
Text description
- A task or request starts the loop.
- The model proposes the next action.
- Stop check: if the task is done, the step or cost budget is spent, or an error occurs, the loop ends with a final answer or a hand-off to a person.
- Permission check: the action must use an allow-listed tool with arguments that pass validation. Irreversible actions first go to a person for approval; a declined action is reported back to the model instead of running.
- The tool runs with least-privilege credentials.
- The result returns to the model as an observation, and the loop repeats.
- Every step — model calls, tool calls, approvals, cost and latency — is recorded in a trace log.
Reference architecture 2
RAG pipeline with citations
Retrieval-augmented generation gives the model passages retrieved from your documents at answer time, so it does not rely only on what it learned in training. A wrong answer can come from retrieval (the right passage never reached the model) or from generation, so the two are measured separately.
Text description
- Ingest: source documents are parsed, keeping their access permissions.
- Chunk: documents are split into passages with source metadata.
- Index: each passage is indexed twice — lexically (by keywords) and as a vector (by meaning).
- Retrieve: for each question, candidates come from both indexes, filtered to what the person asking is allowed to see.
- Re-rank: candidates are re-scored and only the most relevant passages are kept.
- Generate: the answer cites the passages it used, or the system declines when the evidence is missing.
- Evaluate on every change: did the right passages come back, and is each answer faithful to the passages it cites?
Reference architecture 3
Evaluation harness
The harness that turns “it seems better” into a measurement. In our design it comes first: it is built before the system it measures, and handed over with it.
Text description
- Test set: versioned cases, with a held-out split that is never used for tuning.
- Runner: executes the system under test with fixed, recorded settings, repeating runs to measure variation.
- Graders: exact checks wherever possible; an LLM judge is used only after its agreement with human labels has been checked.
- Results store: scores, settings and full run logs, so every number can be traced.
- Report: scores with uncertainty and an analysis of failures.
- Failures found in reports are added to the test set.
Capability matrix
What each kind of system would include
| Kind of system | Model API layerstructured output, retries, fallbacks | Retrievallexical + vector index, re-ranking | Tools & permissionstyped tools, scoped credentials | Human reviewapprovals or exception queue | Evaluation suitetest set, graders, CI gate | Tracing & monitoringfailures, cost, latency | Provenanceversioned prompts, data, model IDs |
|---|---|---|---|---|---|---|---|
| LLM features & applicationsLLM application developmentPlanned | Always | When needed | When needed | When needed | Always | Always | Always |
| Tool-using agentsAI agent engineeringPlanned | Always | When needed | Always | When needed | Always | Always | Always |
| RAG & knowledge systemsRAG & knowledge systemsPlanned | Always | Always | Rarely | When needed | Always | Always | Always |
| Automation & document workflowsAI automation & data workflowsPlanned | Always | When needed | When needed | Always | Always | Always | Always |
| Evaluation harnessesLLM & model evaluationPlanned | Always | Rarely | Rarely | When needed | Always | Always | Always |
| PrototypesPrototypes & proofs of conceptPlanned | Always | When needed | When needed | When needed | Always | When needed | Always |
- Always
- Part of our default design for this kind of system.
- When needed
- Included when the requirements call for it.
- Rarely
- Not usually part of this kind of system.
- Planned
- Being designed. Not yet offered — register interest via Work with us.
Related reading
Research and guides behind this practice
Guide ·
A practical checklist for shipping a RAG system
What to have in place before a retrieval-augmented generation system meets real users: a retrieval evaluation set, structure-aware chunking, hybrid search, verifiable citations, permission-aware retrieval, a freshness plan and production monitoring.
Analysis ·
How to read an AI benchmark table without fooling yourself
Eight questions to ask before you trust a row on a leaderboard: who ran it, which version, under what settings, whether the benchmark still separates models, whether the model has seen it, what the grader checks, how big the noise is, and which harness produced it.
Explainer ·
What counts as an AI agent? A working definition for engineers
A definition you can check against code: a model, a set of tools, a loop, state carried between steps and an explicit stop condition — plus how to tell when a fixed workflow is the better engineering choice.
Paper breakdown ·
ReAct: interleaving reasoning and actions in language models
A technical reading of the 2022 paper that had language models alternate between written reasoning and tool actions — how the thought–action–observation loop works, what it improved on question answering and interactive tasks, and where it failed.
Paper breakdown ·
Retrieval-Augmented Generation: pairing a language model with a searchable memory
A technical reading of the 2020 paper that introduced RAG models — how a dense retriever and a sequence-to-sequence generator are trained together, the difference between RAG-Sequence and RAG-Token, and what the results do and do not show.