Evaluation is the component teams skip when a deadline compresses, on the reasoning that it can be added later. It is also the component that determines whether you can safely change anything after go-live — which makes it the wrong thing to defer.
What a harness actually consists of
- A golden dataset. 100–200 real questions with known correct answers and known source passages.
- Scorers. Groundedness, citation accuracy, retrieval recall, refusal rate — some deterministic, some model-graded.
- An adversarial suite. Prompt injection attempts, out-of-scope questions, ambiguous questions and questions whose answer is genuinely absent.
- A runner. Executes the suite against a deployment and produces comparable scores across runs.
- Gates. Thresholds wired into the pipeline that block a release on regression.
Where the two weeks actually go
| Component | Effort | Notes |
|---|---|---|
| Golden dataset | 4–5 days | The expensive part. Needs subject-matter expert time, not engineering time. |
| Scorers | 2–3 days | Mostly assembling existing libraries and calibrating the model-graded ones. |
| Adversarial suite | 1–2 days | Largely reusable across projects once written. |
| Runner and reporting | 2 days | Straightforward engineering. |
| Pipeline gates | 1 day | Threshold configuration and a failing build. |
Roughly ten working days, with the dataset dominating and requiring the least engineering.
Note where the cost sits. The dataset is the bulk of the effort and it is subject-matter expert time, not engineering time — which is exactly why it gets deferred, because that time is harder to schedule.
Why the second one is much cheaper
Scorers, runner, adversarial suite and gates all carry over. What does not carry over is the golden dataset, which is domain-specific by definition. That is why the second and subsequent systems land around two days rather than two weeks.
The reusable eighty per cent is the engineering. The irreducible twenty per cent is knowing what a correct answer looks like in your domain.
What it buys you
- The ability to change prompts without a week of manual spot-checking
- The ability to evaluate a new model in an afternoon instead of arguing about impressions
- A defensible answer when your risk committee asks how you know it works
- Early warning when retrieval quality degrades as the corpus grows
That last one is underrated. Retrieval quality decays as a document corpus expands, and without measurement the decline is invisible until users stop trusting the system — at which point regaining trust costs far more than the harness would have.
Book the subject-matter expert time for dataset construction in the same week as requirements workshops, while they are already engaged. Trying to get that time back later, once the build is underway, is where most harness projects die.
The comparison worth making
Two engineer-weeks plus a few days of expert time, against one production incident where an AI system gave confident wrong answers to customers. The incident costs more in remediation alone, before counting the credibility that does not come back quickly.
We have not yet met a team that regretted building the harness. We have met several that regretted deferring it.
Building an AI system without measurement?
We can stand up the harness alongside your build, and hand it to your team with the dataset construction process documented.