Two teams, one dataset, no conversation
On a health insurance engagement last year I watched two groups build the same thing eleven floors apart. The platform team had a set of two hundred questions with expected behaviours, run on every merge. The compliance team was assembling a validation pack: representative cases, expected outcomes, evidence of testing. Same content, different filenames, no shared owner.
Nobody was being obstructive. The engineers assumed compliance wanted paperwork they could not produce, and compliance assumed the test suite was implementation detail beneath their remit. It took one meeting to establish that the regulator’s question and the pipeline’s question were the same question asked in different vocabulary.
Treat golden datasets the way you treat test evidence
In every other regulated engineering discipline, tests are evidence. Nobody validates a payments change by describing their intent in prose; they show the test suite, the run, and the result against a threshold agreed in advance. AI systems get an odd exemption from this expectation, mostly because the evaluation practice grew up inside research rather than assurance.
Dropping the exemption is largely an administrative act. The dataset needs a version, an owner, a documented rationale for its coverage, and a retained record of each run against a release. Nothing about that list requires new tooling. It requires deciding that the artefact is official.
Write the thresholds down before you measure
The most damaging habit in evaluation is setting the pass mark after seeing the score. It is rarely dishonest — it usually sounds like a reasonable engineer saying that ninety-one per cent is clearly fine for this use case. But a threshold chosen after the fact provides no assurance to anyone, and an auditor will spot it immediately by comparing document dates.
Agree the numbers with the business owner while the design is still being argued about, and record who agreed. Some of them will turn out to be wrong. Changing a threshold with a stated rationale and a date is a governed act; discovering it was never written down is a finding.
A threshold agreed after the result is published is not a threshold. It is a description of what happened, and it protects nobody.
The cases that matter most are the refusals
Most golden sets over-index on questions the system should answer well, because that is what the product demo needs. The cases that decide a supervisory conversation are the opposite: what the system must decline, hedge, or escalate. Out-of-scope requests, adversarial phrasing, questions the user is not entitled to ask, situations where the correct output is an admission of uncertainty.
We aim for roughly a third of any suite to be non-answers, and we source them from the people who deal with the consequences — complaints handlers, clinical safety officers, the fraud desk. They produce better adversarial cases in an afternoon than an engineering team does in a fortnight, because they have read the incidents.
Start with fifty cases this week
The reason evaluation suites do not exist is almost never disagreement about their value. It is that the imagined version has two thousand cases and a labelling budget, so it stays imagined. Fifty cases, in a file, in the repository, run on every release, with fifteen of them refusals, will tell you more than the plan for two thousand.
Then let it grow from production. Every incident, complaint and “why did it say that” message becomes a case, and the suite starts to encode what your organisation has actually learned. Eighteen months in, that file is the most valuable governance asset the programme owns, and none of it was written for the auditor.
The practical claim
If your evaluation suite is versioned, owned and signed off, it is already most of your evaluation evidence. If it lives in a notebook on someone’s laptop, you will write that documentation twice — once for the engineers and once for the auditor.
Talk to our governance team