How to design an AI evaluation around a real task

An evaluation begins when you can describe the job without using the phrase “answer well”. A useful task definition says what arrives, what the system must do, which outputs count as success, and what failure costs. Without that definition, even a polished benchmark may measure an attractive proxy instead of the work your users need done.

This article gives a concrete way to turn a product workflow into test cases. It is part of the model benchmarking guide; its focus is task and dataset design, not statistical analysis or metric selection.

Write a task contract

Take a support-ticket classifier as a running hypothetical example. A vague contract is “classify tickets accurately”. A testable one might say: given a ticket body and a fixed set of labels, return exactly one allowed label, or “needs review” if the evidence is insufficient. The system must not infer private facts absent from the ticket. A misrouted urgent safety report is more serious than a routine billing ticket labelled as general support.

Record five parts:

1. Input boundary: what data the model sees, including system instructions and retrieved context.
2. Output contract: format, required fields, allowed labels, abstention or escalation behaviour.
3. Success conditions: observable properties of a successful answer.
4. Failure classes: errors that matter, with severity and examples.
5. Operating context: who uses the output, how quickly, and what happens next.

This is your evaluation specification. It should be understandable to a product owner, engineer and reviewer, not only to the person who wrote the prompt. When the job has multiple stages, evaluate each stage and the full workflow. A classifier may be right but still break a service by returning malformed JSON. A retrieval answer may be factually accurate but cite a document the user cannot access.

Sample the work, not the convenient data

Begin by mapping the task into meaningful conditions. For support tickets, these could include clear examples, ambiguous wording, multiple issues, misspellings, urgent reports, rare categories and cases where the correct action is escalation. Include the distributions that matter for deployment, but deliberately oversample rare high-impact events for diagnosis. Keep the operational prevalence visible so you do not mistake a stress test for a production average.

Draw examples from several sources where possible: domain-authored cases, appropriately approved historical interactions, edge cases written by reviewers and synthetic variants used to probe specific behaviours. Synthetic examples are useful for controlled coverage, but they do not establish how often a condition occurs in the real world. Label their origin. Do not manufacture an apparent production distribution from a hand-written set.

Create a dataset row with an immutable ID, input, expected result or rubric, tags, provenance and version. For every item, record why it belongs in the suite. Remove duplicates and near-duplicates where they inflate apparent sample size. If one user or document produces many examples, group them so a random split does not put almost identical cases on both sides.

Make labels usable

A label is only useful if competent people can apply it consistently. Write a short labelling guide with definitions, boundary rules and examples. “Urgent” might mean that a service-level response is required within a defined time, not merely that the user uses emphatic language. If two labels are legitimately possible, represent that ambiguity rather than forcing a false single answer.

Have two reviewers label an initial batch independently. Compare disagreements, resolve what the task contract means, and update the guide before labelling the larger sample. Track unresolved cases and adjudication. Agreement does not prove that a label is correct, but disagreement shows where the instruction or task boundary needs attention. For open-ended outputs, record acceptable alternatives or score dimensions rather than pretending there is one canonical sentence.

Separate development from final evaluation

Use a development set to improve task wording, prompts, tools and graders. Keep a holdout for the final comparison. Split at the right unit: if multiple tickets come from the same customer incident, keep that incident in one partition; if documents from the same template share boilerplate, group by template or source. Otherwise near-duplicates can leak across the split and make generalisation look better than it is.

For a time-sensitive task, a chronological split can be more informative: tune on older cases and test on later ones. For a changing product, use a frozen holdout plus a rolling set of recent incidents. Do not repeatedly tune against the holdout. Every time you use it to make a prompt change, log that use; after enough iteration, retire it into development and create a fresh final set. The concerns around overlap and benchmark familiarity are explored in datasets, holdouts and contamination.

Build a benchmark matrix

A compact matrix makes omissions visible before you spend on runs. For each slice, record the expected behaviour, risk, source, number of cases and coverage owner. For the ticket classifier, a matrix might include all common labels, each escalation path, mixed-intent tickets, empty or malformed input, hostile instructions embedded in a ticket, and out-of-scope requests. These are design prompts, not reported experimental findings.

Do not treat every slice as equally frequent or equally important. Keep two views: a deployment-weighted summary to estimate ordinary use, and a risk-focused view for rare but consequential failures. If you only publish the weighted average, low-frequency safety failures can disappear. If you only publish the stress-test score, readers may overestimate normal-case difficulty. State how cases are weighted and why.

Decide when the set is large enough

There is no magic number of examples. A set needs enough cases to distinguish the outcomes that would change your decision. If two candidates differ by only a small amount, a few dozen examples may not resolve the question. If one candidate violates a strict schema on half of the cases, a small sample may expose the issue quickly. Estimate sample needs from the expected variation and decision margin, then add more cases where uncertainty remains high.

Use a pilot run to find broken labels, missing cases and scoring bugs. A pilot is not a result to promote as a final estimate if you used it to modify the system. Keep the pilot data and the final set versioned separately. When evaluating rare categories, inspect counts as well as percentages: a 100 per cent pass rate on two cases is weak evidence.

Make every case auditable

Store the exact input presented to the system, expected answer or rubric version, model and prompt identifiers, run timestamp, raw output, parsed output and score. Note any redaction or transformation. Hash or otherwise identify immutable test assets when practical. If a case is withdrawn because its label is wrong, preserve the old record and explain the correction rather than silently rewriting history.

Protect sensitive examples with access controls and minimise retained information. A benchmark should not become a shadow copy of customer data with indefinite retention. If a task depends on a particular document or policy version, pin it in the dataset or record a stable reference. A change to source material changes the task, so it belongs in the evaluation history.

Example: a small first version

Suppose a team wants to route product-support tickets. A first suite might have 120 cases: 70 common cases sampled from approved history, 25 difficult mixed-intent cases written by support specialists and 25 escalation cases. Each item has one primary label, a secondary “escalate” flag and tags for language, urgency and ambiguity. The team reserves 30 cases as a holdout, grouped by incident so related messages stay together. This distribution is entirely illustrative; it is not a recommended universal sample size.

The team runs a pilot on the 90 development cases, discovers that “account access” and “billing access” are inconsistently labelled, fixes the guide and relabels affected items. It then freezes a test version. The scorecard can report ordinary weighted accuracy, escalation recall, malformed-output rate and results for mixed-intent examples. This is more actionable than saying only that the model scored 88 per cent.

A launch checklist

Before comparing models, answer these questions:

• Does the task contract define input, output, success and failure?
• Are common, difficult and high-risk conditions represented?
• Can reviewers apply labels consistently, and are disagreements preserved?
• Are related examples grouped appropriately across splits?
• Is the holdout protected from prompt tuning?
• Are origins, redactions and dataset versions documented?
• Can another engineer reconstruct the exact evaluated configuration?

If several answers are “no”, fix the evaluation design before interpreting model differences. A smaller well-defined suite can be more useful than a large unlabelled dump. Once the cases and labels are solid, choose fit-for-purpose metrics and rubrics, control the comparison settings, and report the uncertainty honestly.

Evidence for the evaluation design

HELM: Holistic Evaluation of Language Models documents scenario coverage, multiple evaluation dimensions and standardised adaptation. It supports making the tested task and configuration explicit; the sampling and release choices above remain design recommendations for your workload.