In traditional software, the specification is a document. Engineers write code that satisfies it, tests check the match, and the spec is the source of truth.
In AI engineering, the specification is the evaluation set: a collection of inputs and expected outputs, sometimes with rubrics, that defines what good looks like. The model is the implementation, judged against it. That sounds like a small change. It turns the engineering process upside down.
It matters most in deployment work, because the eval set decides whether validation succeeds, and validation decides whether the engagement does. If the evals test for things the customer doesn't care about, or miss things they do, a system that passes will still fail in production. Hamel Husain has been blunt about this: failing AI products almost always share one root cause, the absence of a working evaluation system.
What a working eval set looks like
- Built from real customer data. Synthetic data lies. It's cleaner, narrower, and free of the edge cases real data throws at you. The fight for data access is part of the job. Duck it to ship faster and you'll lose the engagement.
- Big enough for signal, small enough to iterate. One hundred to a few hundred examples for the first version. Ten thousand is too many: grading gets expensive and the cycle gets slow enough that people stop looking at the results.
- Specific to this customer's outcome. Public benchmarks help you pick a base model. They tell you nothing about whether your system, for this customer, will produce the business result they paid for.
- Weighted toward hard cases. The customer's experts know which cases are hard. Ask them. A system that passes the easy ones and fails the hard ones isn't ready.
- Labeled by people the customer trusts. Vendor-side labels carry no weight with the customer. Give their experts an interface that lets them grade two to five examples a minute, or the labeling won't happen.
- Versioned. The set will change, and you need to know whether a better score came from a better system or a moved bar.
LLM-as-judge: calibrated or worthless
The hardest layer is the middle: cases where the right answer can't be written as an assertion, but human grading doesn't scale. The standard tool is an LLM-as-judge, a separate model call that grades outputs against a rubric.
It works, with conditions. Judges have predictable position, verbosity, and self-preference biases, and they only earn their keep when calibrated against human labels on your own data. Without calibration you're laundering noise: the judge produces scores, the team acts on them, and nobody knows how the scores relate to the customer's real quality bar.
Calibration is tedious and simple. Have the customer's experts label fifty to a hundred outputs, have the judge label the same set, and measure agreement. Above roughly eighty percent, you have a working judge for that rubric on that data. Below it, revise the rubric or prompt until agreement climbs. Recalibrate whenever the underlying model changes.
Expect what Shreya Shankar's research calls criteria drift. People can't fully define their quality criteria up front. The first rubric is always wrong, and grading real outputs is what clarifies it. Draft, grade fifty examples with the customer's expert, revise, repeat.
A contract-review eval, in numbers
On a contract-review deployment I ran for a mid-market commercial real estate firm, the customer's deputy general counsel pulled forty-two real contracts from the previous six months to start the eval set. Her grading capacity was about five contracts an hour, and we needed hundreds of grades a week as the prompts changed. So we calibrated a judge.
On the hard-fail criterion (did the system miss a material clause?), judge-human agreement reached ninety-four percent after two rubric revisions, high enough to trust. On a softer quality criterion, agreement stalled at seventy-one percent, so we kept that one manual until the rubric settled, and a third pass took it to eighty-six.
Over three months the set grew from 42 cases to 131. Every case that surprised her went in, labeled the same day. By month three it had become the firm's internal reference for what a good review looked like. We never planned that, and it turned out to be the most valuable thing the engagement produced.
Eval pass rate is the new NRR
Evals do their best work as the signal the whole build cycle climbs against, not just as a gate. And as contracts move toward paying for outcomes, the eval bar becomes the revenue line. Milos Mandic puts it neatly: "Eval pass rate is the new NRR." The rate at which the system's work clears the customer's bar is what renewal and expansion hang on.
The bar isn't universal. It depends on the cost of an error. In aerospace a defective part is catastrophic, so the bar is 100 percent. For a wholesaler, a wrong shipping address costs fifty cents, so 95 percent is fine. Define the cost of error first, and it tells you what "good enough" means.
There's a customer-side version too. Eno Reyes, Factory's CTO, calls it agent readiness: how many deterministic validation loops (linters, type checkers, tests, security scans) exist in the customer's environment. An agent can only be trusted with as much autonomy as its work can be checked. The evals you build with a customer are the start of their agent readiness, and their agent readiness caps how much autonomy you can responsibly hand them.
My one concrete recommendation: after scoping, spend the first week of validation on nothing but the eval set. No production code, no integration. Build the evals, run the baseline, and show the customer the gap between where the system is and the bar they're paying for. If the gap is small, the engagement is short. If it's large, you both know it now, before the late-stage scope fight that sinks so many projects.
Thinking about moving into this work?
I wrote a free guide for people considering forward deployed engineering: what the 4.6x growth figure means, how to tell a real FDE role from a renamed sales job, and a 30-day plan that ends with your first piece of proof.