Building a portfolio that proves AI engineering skill

What a portfolio has to prove

Whoever reviews your work — a hiring manager, a tech lead, a prospective client — is trying to answer a handful of questions. Can you turn a vague problem into a defined one? Can you build something that works? Can you tell how well it works, and when it does not? Can you explain the trade-offs you made? Could someone else run what you built?

A demo answers the first two at best: a chat interface over some documents, a screenshot, a link. The projects that stand out answer the last three, because measurement, judgement and reproducibility are what separate engineering from prompting.

Fewer projects, more depth

Two or three projects with real evaluation will say more than ten tutorial reproductions. A strong portfolio project usually has:

• A real problem with a real user, even if that user is you. Searching the minutes of a public body you follow, triaging issues for an open-source project you use, extracting fields from public documents in a domain you know — specific beats generic.
• A baseline. The simplest thing that could work: keyword search, a regular expression, a single prompt. Without it, no improvement means anything.
• An evaluation set you built and a metric you chose before iterating.
• A log of iterations with numbers: what you changed, what moved, what did not.
• Failure analysis: the categories of error that remain, with examples.
• Cost and latency, measured, because every real system has a budget.

The numbers in the second column are placeholders for your results, not targets.

Evaluation is the differentiator

Evaluation is where a project shows the most judgement for the least code. It is also the part a reviewer cannot infer from a demo.

• Start small and hand-label. A few dozen carefully labelled examples you understand beat thousands you have never read. Grow the set as you find new failure modes.
• Hold out a test split. If you tune prompts on the same examples you report, your numbers measure memory, not generalisation.
• Report the sample size and the uncertainty. A difference of a few points on a small set may be noise; say so. Our piece on reading benchmark tables includes a short function for a confidence interval.
• Compare approaches, not just versions. Baseline versus your system, and ideally one serious alternative.
• Show the errors. An error table — category, count, example, likely cause — demonstrates more understanding than any headline score.

For retrieval projects specifically, our RAG shipping checklist covers recall@k, MRR and how to build the labelled set.

Write-ups that engineers actually read

A good write-up lets a busy reviewer understand the project in two minutes and verify it in twenty. A structure that works:

1. Problem and user. One paragraph: who has the problem, and what they do today.
2. Result first. What works, how well, measured on what.
3. Approach. The architecture in a short diagram or list, and why you chose it over the alternatives.
4. Evaluation. Dataset, how it was labelled, metric, baseline, results table with sample sizes.
5. What did not work. Negative results are evidence of judgement, not weakness.
6. Limitations and next steps. Where it breaks, and what you would do with more time.
7. How to reproduce. Exact commands.

Write in plain language and keep claims proportionate to the evidence. "Worked well on my 80-question test set" is credible; "production-grade" for something that has never served a user is not.

Make it reproducible

Reproducibility is one of the easiest ways to show professional habits.

• Pin everything: a lockfile for dependencies, the language version, and ideally a container or dev-environment file.
• Record exact model identities: the full model identifier or snapshot name for hosted models, and the revision for open-weight ones, plus sampling settings.
• Keep prompts in version control as files, not strings buried in notebooks.
• Document data provenance and licensing. Do not commit data you are not allowed to redistribute; provide a script that fetches it instead.
• Acknowledge non-determinism. Hosted models may not return identical outputs run to run. Cache raw model responses so reviewers can re-run your analysis without paying for API calls, and say which results were recomputed.
• One command to the results table — make eval or equivalent — that writes the numbers you quote.
• Never commit secrets. Read keys from environment variables and ship an example environment file with placeholder values.

Engineering hygiene counts too: tests for the non-model parts, error handling around API calls, logging, a README that starts with what the project does. A commit history that tells the story of your iterations is itself part of the write-up.

If you freelance: scope first, estimate honestly

Much of what goes wrong in a freelance AI project can be prevented while scoping it. A few practices protect you and the client.

Start with the decision, not the model. Ask what decision or workflow the system will change, who will use it, what data exists and who owns it, and what constraints apply — privacy, where data may be processed, integration points. Sometimes the honest answer is that the problem does not need AI, and saying so builds more trust than a proposal.

Write acceptance criteria as an evaluation. Agree on a test set and a threshold before building: "on these 100 labelled cases, at least this share handled correctly, with these failure types flagged for review." It turns a subjective "is it good?" into a shared, checkable definition of done.

Phase the work. A short, paid discovery or feasibility phase; then a prototype measured against the agreed test set; then hardening and handover. Each phase ends with a decision point where either side can stop.

Estimate in ranges with named assumptions. "Two to four weeks, assuming data access in week one and fewer than N document formats" is honest; a single number is a guess dressed up as a promise. List the unknowns — data quality, access, rate limits, stakeholder availability — and re-estimate after discovery.

Be explicit about what you will not promise. Model behaviour on data you have not seen, accuracy beyond what the agreed evaluation shows, or fixes to upstream data you do not control.

Hand over something maintainable: documentation, the evaluation harness, a runbook for common failures, and a clear list of what to monitor.

Ask before publishing client work. Many engagements cannot be shown. With permission you may be able to publish an anonymised write-up; otherwise, rebuild the method on public data and publish that instead.

Before you share a project

• The README states the problem, the result and how it was measured, in the first screen.
• There is a baseline and at least one comparison.
• Sample sizes and uncertainty are stated next to every number.
• Failures are shown, with examples.
• Model identifiers, settings and prompts are pinned and versioned.
• One command reproduces the results.
• No secrets, no data you may not redistribute.
• Every claim is one you would be comfortable defending in an interview.