Turn an AI workflow into an evaluation dataset

An evaluation dataset is a set of questions you are prepared to answer about your system. Collecting a hundred convenient prompts is not enough if they all test the same easy behaviour. Good task design starts with the decisions users need to make and works backwards to inputs, context, expected behaviour and failure consequences.

Consider a fictional assistant answering staff questions from supplied policy excerpts. Its job is not to recite a policy from memory. It must use the available evidence, cite it accurately and acknowledge when the excerpts do not settle the question. The examples and proportions below are design illustrations, not observed traffic.

Map the workflow before collecting cases

Write a one-sentence objective and identify the boundary of the system under test. Is retrieval included, or will every candidate receive the same excerpts? Does a human review the answer? Can the system act, or only advise? Who uses the output and what happens if it is wrong? These answers determine whether you are evaluating a model response, a retrieval application or a complete agent.

Describe the ordinary journey in steps. A staff member asks a question; the system obtains passages; it decides whether they support an answer; it cites the passages; the user reads or follows the answer. Each step can fail differently. Retrieval can omit the relevant exception even when the model reasons correctly from what it receives. A model can invent an exception despite a correct retrieval. Test both the end-to-end journey and components when you need to diagnose the cause.

OpenAI's evaluation best practices recommends task-specific cases and a mixture of typical, edge and adversarial examples. Treat that as a design starting point: the right mix and acceptance criteria still depend on the actual workflow.

Write a case contract

Each case needs a stable ID, input, available context, expected behaviour, scoring information and provenance. Add tags for task type, risk, language, document family and difficulty where useful. Record the source of the example, permission to use it, any redaction, and the date or version of the underlying policy. Keep sensitive identity information out unless it is essential and appropriately controlled.

A compact example might be:

• ID: policy-014; type: missing evidence.
• Question: “Can a contractor approve this expense?”
• Available excerpt: a fictional policy describes employee approval but says nothing about contractors.
• Expected behaviour: state that the supplied excerpt does not establish contractor authority; ask for the relevant policy or route to an authorised reviewer.
• Critical failure: assert that contractors can approve without supporting evidence.

The expected behaviour is not necessarily a single reference sentence. It can specify acceptable alternatives and forbidden assertions. A case can have more than one valid answer, especially when a cautious clarification is appropriate. Record why the reference is authoritative and ask a domain expert to review ambiguous cases.

Build slices that answer different questions

Start with common traffic: straightforward questions, ordinary document lengths and normal phrasing. Then add slices for missing evidence, conflicting passages, exceptions buried in long text, misleading quotations, unusual terminology and questions that require escalation. Include cases where a refusal would be wrong because the answer is available; otherwise a system could improve its safety score by refusing everything.

Separate frequency from importance. If rare high-risk cases are deliberately oversampled, report that slice on its own. A traffic-weighted aggregate can be useful only if the weights are credible and current. For an exploratory dataset with unknown traffic proportions, report unweighted slice scores and do not call the result a production success rate.

Avoid creating “hard” cases solely by making instructions confusing. Realistic difficulty comes from the task: evidence may be incomplete, an exception may alter the answer, or two sources may conflict. A deliberately adversarial input is useful when it represents a threat you want to test, but label it so readers do not mistake it for ordinary usage.

Sample with a purpose

Production logs can reveal real wording and failure modes, but first establish a lawful and authorised basis for access, minimise personal information and set retention limits. Obtain domain review where the reference answer depends on a policy. Historical examples may overrepresent users who complained or situations your old system handled badly; note the selection mechanism.

Synthetic cases can broaden coverage before enough real examples exist. Label them, review their assumptions and compare them with genuine traffic when available. They can accidentally repeat the biases of the author or model that generated them. Do not use generated reference answers as truth without independent checking.

Create a coverage table rather than a quota alone: each row is a behaviour, risk or user group, and each column records case count, provenance, expected outcome and whether it appears in development or holdout. A small table makes gaps visible. More examples are valuable when they add independent information, not when they merely paraphrase the same underlying document.

Separate development from a decision test

Group related examples before splitting. If several questions refer to the same document or incident, placing variants on both sides can make the holdout look easier than genuinely new material. Keep a development set for prompt and rubric changes, and a restricted holdout for the planned final comparison. The holdout guide explains access controls and when a repeatedly consulted set needs replacement.

Before running candidates, check each case by hand. Can a reviewer determine the outcome from the evidence provided? Does the question accidentally reveal the answer? Is the reference current for the document version? Does the case permit a safe answer when evidence is missing? Resolve ambiguous labels or retain them as explicitly ambiguous cases with a suitable rubric.

Run a small pilot and inspect failures

A pilot is for finding broken cases and scoring rules, not declaring a winner. Run a handful of examples through the full harness, including one expected refusal and one clear answer. Verify that inputs are rendered correctly, references are available to graders, failures and timeouts are captured, and reviewers agree on the intended meaning. Change the development set and its version if you discover defects; preserve a record of what changed.

Then freeze the case manifest and scoring rules for the comparison. Record which cases were excluded and why. Do not quietly drop hard examples because a candidate failed them. If an infrastructure fault makes a case ungradable, report it and apply a pre-agreed retry or exclusion rule consistently across candidates.

The output of task design is a dataset that can support a specific decision, not a claim of universal coverage. Pair it with observable metrics, and revisit the coverage table as the workflow and its users change.

A traceable case review

Before accepting a case, ask a second reviewer to reproduce the expected outcome using only the supplied evidence. For an answerable question, they should point to the passage supporting each material assertion. For an unanswerable question, they should explain what additional information would be required. If a case has two reasonable interpretations, either clarify the task or record both acceptable responses and the conditions that distinguish them.

Give each accepted case a lifecycle state such as proposed, reviewed, active or retired. A retired case can remain valuable historical evidence even when a source policy has changed. Record the reason for retirement and avoid silently replacing its text under the same version. This is especially helpful when an old evaluation is challenged months later: the team can see exactly which policy and expected outcome were in force for that run. Assign an owner for each high-risk slice so that changes in the underlying workflow lead to an intentional review of its cases.

A completed case record

Copy the fields into an access-controlled case store and replace the example with authorised evidence. This filled record makes the boundary inspectable; it does not create an approved real-world case.

Return to the Terra cluster guide · Inspect the four-way results and evidence.