2026-09-23 · 7 min · By Alcott Dube
How to evaluate an ai feature before you ship it
I evaluate automated features with representative test cases, structured failure logs, and a completion metric that counts correction work rather than hiding it.
I evaluate an ai feature by testing representative tasks against explicit acceptance criteria, logging failures by cause and severity, and measuring acceptable completion without substantive human correction. I ship only when that result improves on the existing approach, serious risks have separate release gates, and a limited rollout confirms the benefit in actual use.
Define success before choosing evaluation metrics
I start evaluating ai features with one completed task, not a model capability. For a support-drafting feature, the task might be: produce a reply that answers the customer's question, follows the refund policy, and contains no invented account details. A fluent paragraph isn't the outcome. A reply someone can use is.
Google's People + AI Guidebook connects evaluation with user needs and the consequences of errors. I turn that into a written acceptance rule before touching a prompt. For this example, a reply passes only if its facts are supported, its proposed action is permitted, and the reviewer needs no substantive edits.
I also document the current alternative. That could be a saved reply, a search tool, or a person writing from scratch. I measure its completion time and correction burden on comparable tasks. If generation saves 20 seconds but verification adds 40, I haven't improved the task.
Build a test set from real task variation
I'd start a narrow drafting feature with 160 cases: 80 ordinary requests, 40 ambiguous or incomplete requests, 20 attempts to redirect the system, and 20 situations where a mistake could cause significant harm. Those are starting allocations, not a statistically representative sample or evidence that the feature is safe.
I use permissioned, appropriately redacted examples where available, then write synthetic cases for gaps. I include short messages, contradictory context, missing records, and requests outside the feature's remit. Each case gets an expected behaviour, supporting evidence, and a severity label. For ambiguous requests, asking for clarification may be the correct result.
I reserve 40 cases as a held-out check and use the other 120 during development, keeping coverage across the groups. I don't keep tuning against the held-out cases. Once their details influence a change, I treat them as development material and replenish the check. I label synthetic cases so their results aren't mistaken for observed customer behaviour.
Choose graders that catch consequential mistakes
Anthropic's evaluation guidance recommends defining success clearly and matching evaluation methods to the task. I separate checks that code can settle from checks that need judgement. Required fields, valid record references, and prohibited tool calls belong in deterministic checks. Whether a reply resolves an ambiguous complaint needs a different assessment.
For that judgement, I write a short rubric with observable criteria. 'Good answer' is useless. 'Explains the applicable refund condition without promising an exception' gives a reviewer something concrete to assess. I include examples of a pass, a failure, and a borderline response, with reasons for each decision.
A model grader can help sort a larger batch, but I wouldn't make it the sole release authority for consequential outputs. I check its decisions against human-reviewed examples, especially confident passes on risky cases. When reviewers disagree, I inspect the rubric before averaging their scores. The disagreement may expose an undefined product rule rather than a weak model.
Measure completion without substantive correction
My primary metric is acceptable completion without substantive human correction: the number of tasks that meet the acceptance rule without meaningful repair, divided by all eligible tasks assigned to the feature. I define meaningful repair in advance. Changing punctuation doesn't count; correcting a refund amount or removing an invented promise does.
Suppose 200 eligible tasks enter the feature and 146 produce acceptable results without that repair. The rate is 73 per cent. Timeouts, abandoned attempts, and failed generations stay in the denominator. Otherwise, the dashboard rewards the feature for making difficult cases disappear. A proper clarification can pass when clarification is the specified task outcome.
I don't use this rate alone. I track review time, total handling time, cost per acceptable completion, and severe errors separately. For a drafting feature, a safe handoff may prevent harm without completing the draft, so I record both facts. I report results by case group because an overall average can hide a serious weakness in less common requests.
Log failures so each one suggests a fix
A failure log should explain what to change, not just preserve disappointing outputs. I record the case identifier, input reference, model and prompt versions, retrieved evidence, tool actions, failed criterion, severity, and reviewer reasoning. I restrict access and retention for sensitive material rather than copying entire customer conversations into a shared spreadsheet.
I distinguish the visible symptom from its likely cause. An incorrect refund amount might come from missing evidence, stale retrieval, faulty reasoning, or an application calculation. Those need different fixes. Adding another prompt instruction won't repair a stale policy document, and changing models won't necessarily repair the calculation.
I give each investigated failure a proposed fix and a regression case. Then I rerun the affected group and the wider suite. A stricter refusal rule might reduce unsupported promises while rejecting legitimate requests. The log needs to show both movements. I prioritise by consequence and frequency, not by which output looks most embarrassing in a demonstration.
Set release gates before reading the results
I write release gates before the final evaluation. For an illustrative low-risk drafting pilot, I might require at least 85 per cent acceptable completion, lower median handling time than the current workflow, and no observed unauthorised disclosures. These are product-specific decisions, not universal thresholds. Zero observed disclosures in a small test doesn't establish zero risk.
I repeat cases when output variation could change the decision. Running each of 40 held-out cases three times gives 120 executions, but still only 40 distinct situations. I report both counts. Repetition reveals instability; it doesn't substitute for broader coverage or justify a precise claim about rare failures.
Anthropic's guidance on building effective agents favours simpler approaches where they suffice. I apply that discipline to release decisions. If a fixed workflow with one generation step meets the need, I don't add autonomous tool selection for novelty. Fewer decision points make failures easier to isolate and consequential actions easier to constrain.
Validate the metric in a limited rollout
Offline results earn a limited rollout, not unrestricted access. I start with a small eligible group, keep consequential actions behind approval, and preserve the existing route. I define who can stop the rollout and which events require an immediate pause. An unauthorised external action should not wait for a weekly metrics review.
During the pilot, I measure actual correction work rather than relying on approval clicks. People may accept an output after fixing it elsewhere, or approve it without noticing an error. With appropriate permissions, I inspect a sample of completed tasks and compare the recorded outcome with what happened afterwards.
New failure patterns go into the development set, while fresh held-out cases protect the next release decision. I rerun evaluations after changes to prompts, models, retrieval, or tools. If the system cannot reliably separate cases it can handle from cases needing a person, I narrow its scope before increasing exposure.
Questions people ask
How many test cases do I need to evaluate an ai feature?
For a narrow feature, I'd begin with roughly 100 to 200 varied cases to expose common failure patterns. That is a development starting point, not a statistical safety claim. Higher-consequence tasks need deeper coverage and a justified sampling plan.
Can I use another model to evaluate generated answers?
I use model graders for criteria that have a clear rubric and examples. I validate their decisions against human review and deterministic checks, rather than treating agreement between two models as proof of correctness.
What is the best metric for an ai automation feature?
I start with acceptable completion without substantive human correction, measured across all eligible assigned tasks. I pair it with handling time and separate severe-error gates, because a high completion rate can still conceal unacceptable harm.
When should I rerun my evaluation test set?
I rerun it when a model, prompt, retrieval source, tool, or product rule changes. I also add regression cases after investigated failures, while keeping a separate held-out check for release decisions.