Evals are the new unit tests
If you cannot say whether last week's prompt change made the system better or worse, you are not engineering. The smallest eval suite worth having.
Ask a team how they know their AI feature works and you get one of three answers.
"It seems good." "We tried a few questions." Or, occasionally, a number — and that third team ships far more confidently than the other two, because they are the only ones who can tell whether a change helped.
An eval suite is the difference between an AI system and an AI demo. It is also, in our experience, the thing most likely to be missing from a project that has been stuck at ninety percent ready for four months.
Why the usual testing instinct does not transfer#
Software tests are deterministic. Same input, same output, pass or fail.
Language models are not, and this breaks the reflex two ways. First, a correct answer can be phrased fifty ways, so string equality is useless for anything generative. Second, everything is coupled: change the prompt to fix one complaint and you can silently break eleven answers that were fine, because there is no compilation step to catch it.
So evals are not unit tests in mechanism. They are unit tests in function: a suite you run on every change, that tells you whether you moved forward or backward.
The smallest suite worth having#
You can build this in about a week. Most of that week is not engineering.
A golden set of fifty questions. Real ones. Pull them from Slack, from the analyst request queue, from the questions people actually ask in Monday meetings. Not the ones that demo well — those are already right.
Include the awkward ones deliberately: questions with ambiguous entities, questions that span a date boundary, questions the system should refuse because the data does not exist. A suite of only answerable questions will not tell you whether your refusal behaviour works, and refusal behaviour is where trust is won.
A correct answer for each. This is the expensive part, it needs a human who knows the business, and there is no way around it. Budget two or three days. It is the most valuable artefact the project will produce, and it outlives every model you use.
A scoring method per question type.
| Question type | How to score |
|---|---|
| A number | Exact match, or within a stated tolerance |
| A list | Set overlap — precision and recall against the expected set |
| A short factual answer | Substring or entity match |
| A generated explanation | LLM-as-judge against a written rubric |
| A refusal | Did it refuse, and did it say why |
A judge, where you need one. For anything free-form, a second model scores the answer against a rubric. The rubric has to be specific — "is this correct, grounded in the retrieved context, and complete" beats "is this good" by a distance. Calibrate the judge once against fifty human-scored examples so you know how far to trust it, and re-check that calibration when you change judge models.
A number, tracked over time. One overall score, plus a per-category breakdown so you can see which kind of question regressed. Store the history. The trend line is what makes it useful.
Where to run it#
Three places, all of them cheap once the suite exists.
On every change. Prompt edits, model swaps, retrieval changes, semantic model changes. Especially semantic model changes — the analytics engineer renaming a column has no idea they are touching the AI system, and the eval is what tells them.
Before release, as a gate. Pick a threshold. Below it, nothing ships. This is uncomfortable the first time it blocks a release and it is the whole point.
On a schedule, against production. Models change underneath you. Data drifts. A weekly run against the same golden set catches the day your provider's version bump moves your accuracy three points.
Online signals, once you are live#
The golden set tells you about known questions. Production tells you about the rest.
- Refusal rate. Rising means retrieval or coverage broke. Falling to near zero often means the model started guessing.
- Retrieval hit rate. How often the system found something relevant. Failed retrieval is the most common upstream cause of a bad answer.
- Human corrections. Every correction is a free golden-set entry. Capture them and feed them back — this is the highest-value loop in the whole system.
- Thumbs down, read individually. The rate is nearly meaningless. The content is where the next ten test cases come from.
- Cost and latency per answer. Both drift upward quietly as context grows.
Two failure modes to design against#
The suite that only contains easy questions. It will score ninety-five percent and tell you nothing. If your score is not occasionally uncomfortable, your suite is wrong. Deliberately keep a handful of cases you currently fail.
The suite nobody runs. If it takes twenty minutes and a manual setup, it will be run before demos and never before commits. It needs to be one command, in CI, with the result posted where the team already looks.
What it buys you#
Beyond the obvious — knowing whether you are improving — an eval suite buys three things people underestimate.
It makes model choice trivial. Swap the model, run the suite, read the number. What is currently an argument becomes a measurement, and it means you can adopt a better model the week it lands rather than the quarter after.
It makes the system defensible. "We evaluate every release against a hundred-question benchmark and here is the twelve-month trend" is a very different conversation with a risk committee than "it seems good".
And it turns disagreements into test cases. When someone says the assistant is wrong, the reply is "add it to the golden set with the answer you expected". Complaints become coverage.
That last one changes the culture around the system more than any of the engineering does.
Is this the shape of your problem?
We build agentic AI and analytics pods — analytics engineers, AI engineers and decision scientists, with a lead who has done this for thirteen years. Start with a two-week Readiness Check.