"We tested it" is the least informative sentence in enterprise AI. Here is exactly what we mean by it, so you can hold us and anyone else to it.
The golden set
Before an agent exists, we sit with the people who do the job today and collect two hundred real cases with the outcome a good colleague would produce. Claims that should be paid, claims that should be escalated, tickets that need a human, documents that belong to the wrong desk. Each case has an expected outcome and a note on why.
The golden set is owned by the process owner, not by us. It is versioned alongside the agent. When the business changes a policy, the golden set changes first and the agent is re-run against it.
Four kinds of evaluation
- Outcome accuracy. Did the agent reach the expected result? Measured per case and per category, with a confusion matrix the process owner can read.
- Process compliance. Did it call the right tools in the right order, ask for approval when the threshold said so, and stop when confidence dropped? Traces are checked, not just answers.
- Safety and policy. A red-team suite: prompt injection through documents and e-mails, attempts to extract other customers' data, requests outside the agent's remit, tone and language rules. Every finding becomes a permanent test.
- Cost and latency. Tokens and time per case, by model route. A change that improves accuracy by one point and triples the cost gets discussed, not merged.
Thresholds, written down
Every agent has a release threshold per category. For a billing agent that might be 97% outcome accuracy overall, 100% on "must escalate" cases, zero red-team failures, and a cost ceiling per ticket. The thresholds are agreed with the business before the first evaluation runs, so nobody moves them afterwards to make a release fit.
The sign-off ritual
A release goes to production when three things are true:
- The evaluation report for this exact version is green against the thresholds.
- A named person on the client side has read the report and approved it. Their name is in the release record.
- The rollback plan is one click: the previous version, with its own green report, is still deployed and can take traffic.
Nothing reaches a customer without a passing evaluation and a named human sign-off. Not for a hotfix, not on a Friday.
In production, the test never stops
Sampled conversations are scored every day against the same rubric. Drift in a category triggers a review, not an alarm. New edge cases found in production are added to the golden set, so next month's release is tested against this month's reality.
This is a lot of plumbing. It is also what lets a regulated business say yes to an agent that touches real customer data.