Design an LLM evaluation around tasks your users actually do

“Give helpful answers” is a product ambition, not an evaluation specification. A useful task describes what the system receives, what it may do and what observable behaviour counts as success. Without those boundaries, reviewers can grade different interpretations of the same output.

Start by writing one decision that the evaluation will inform. For a hypothetical support assistant, that might be whether a new configuration can draft policy-grounded responses for operator review. It is a narrower decision than allowing the assistant to issue refunds automatically.

The task design below is a recommended working method. All cases are illustrative; they do not describe customer records or measured performance.

Write the decision contract first

Name the intended user, action and consequence. “A support operator uses the answer to decide the next step on a return request” tells you more than “customer service chatbot”. It identifies the information required and the kind of error that could mislead someone.

State the permitted outcomes. The assistant may answer from supplied evidence, ask a necessary question or escalate an unsupported request. Asking a question is sometimes successful behaviour. Treating every non-answer as failure would encourage guessing.

Set exclusions too. If the first release cannot access payment tools, task cases should not assume it can verify a refund. They may test whether it truthfully describes that limitation. This keeps the evaluation aligned with an achievable product boundary.

List the conditions that would prevent release regardless of average quality. For example, an invented action confirmation may require remediation even if most answers read well. The scoring guide explains how to keep such conditions separate from stylistic preferences.

Choose the unit of work

Decide whether a case is a single request, a complete conversation or a stateful workflow. A single-turn answer test can isolate grounding in a supplied policy. A conversation test can check whether the assistant remembers a clarification. A workflow test can check the state after tool actions.

Do not substitute one for another without saying so. If deployment gives the assistant previous messages but the evaluation removes them, the task has changed. Conversely, giving the model the entire final conversation can reveal information it would not yet know at the scored turn.

Write the observation point explicitly: “Evaluate the next assistant response after the user provides the order number.” That sentence helps the dataset curator, runner and reviewer assemble the same task.

For systems that act, specify the initial state and allowed transitions. A correct final response with the wrong database change is not a completed task. Use the agent evaluation article when the unit of work includes actions.

Give every case a usable schema

A practical case record needs a stable ID, task family, input, context snapshot, expected behaviour, scoring evidence, risk labels and provenance. Keep model input separate from reviewer-only information, including reference answers and eventual outcomes.

Consider an illustrative case called RETURN-014. The user asks whether an item can be returned. The supplied policy allows eligible unused items within a stated window, but the order record lacks a delivery date. Expected behaviour is to request the missing date or route to an approved source, without declaring eligibility.

The reference should explain why clarification is necessary. A single “gold” sentence is insufficient because several natural phrasings may be acceptable. Record required content, forbidden claims and allowable next steps. This gives graders a behavioural specification rather than a writing imitation task.

For this case, relevant labels might include missing information, policy grounding and no action permission. Those labels allow later analysis by failure mechanism. Avoid labels that merely restate the answer or inadvertently become input features.

Build a coverage matrix

Use two axes: the work people attempt and the conditions that make it difficult. For the assistant, work families could include eligibility questions, return instructions, order-status explanations and unsupported requests. Conditions could include complete records, missing fields, contradictory evidence and irrelevant user instructions.

Choose combinations deliberately. You do not need every possible cell, but you should know which cells are empty. Record the reason for each important omission, such as an unsupported product feature or unavailable source data.

Behavioural testing offers a useful precedent. CheckList organises tests around linguistic capabilities and test types to expose failures beyond a held-out accuracy score. Its core lesson for this workflow is to specify behaviours worth challenging, rather than relying only on an aggregate. CheckList paper.

Separate a representative sample from a challenge set. The representative sample estimates performance on the selected traffic population. The challenge set probes deliberately difficult or consequential behaviour. A challenge-heavy collection should not be described as a typical production success estimate.

Include negative and ambiguous cases

Add cases where the right action is to decline, ask or escalate. A supplied document may not answer the user's question. An order may belong to another account. Two policies may conflict because their effective dates differ. Write what the system should do with each uncertainty.

Make the ambiguity intentional. If reviewers cannot determine the correct action because the case itself is broken, label it unscorable and investigate. Do not silently turn curator confusion into a difficult reasoning benchmark.

Use contrastive cases to check whether a relevant change alters behaviour. In one case, a policy condition is satisfied; in a paired variant, it is not. Keep the pair in the same data split. Otherwise a near-duplicate can cross from prompt development into final evaluation.

Include irrelevant changes as well. Rewording a polite request should not change eligibility. The exact invariant depends on the task, so state it with the case. These targeted examples help explain why a configuration fails, even when they are not frequency-weighted evidence.

Pilot the specification before collecting more

Ask reviewers to label a small, varied pilot independently. Compare disagreements at the level of required behaviour. If one reviewer rewards an answer while another expects clarification, revise the task definition or reference evidence before expanding the set.

Check that the runner can construct every input without reading the answer fields. Check that the scorer can distinguish an operational failure from an unscorable record. A task specification is incomplete if it leaves these decisions to undocumented conventions.

Choose collection size from the decision's precision and coverage needs rather than a universal minimum. Begin with enough varied cases to test the specification, then plan a larger comparison based on variability, independent groups and important slices. The uncertainty guide explains why repeated outputs are not a substitute for new cases.

Finally, write a one-page task contract containing the boundary, input schema, acceptable outcomes, exclusions, coverage slices and scoring obligations. Attach it to the dataset version. Continue with dataset protection before tuning prompts, and use the pillar guide to connect the eventual result to a release decision.