Designing Evaluation Tasks and Datasets That Reflect Your Workload

The most common evaluation failure is not a bad model. It is a test set that was assembled from whatever was easy to find. Convenience examples or unvalidated synthetic cases can produce a score while leaving workload relevance unresolved. Independently validated synthetic data can provide useful evidence when its provenance, selection and limitations are explicit.

A useful evaluation set is a sample of your workload with a defensible sampling frame, a clear oracle and a labelling protocol. This article sets out how to build one. It assumes you have already named the decision, as described in the pillar guide.

Start with the decision, not the data

Write the decision in one sentence: "Choose the extraction model for supplier invoices, where a missed line item is worse than an extra line item." That sentence tells you what to sample, what to score and which errors matter.

Then write the acceptable error. A 2% line-item miss rate may be fine for a low-value supplier and unacceptable for a regulated one. The threshold determines whether you need 100 tasks or 1,000, and whether a small pilot is enough to decide. Without a threshold, every result is ambiguous.

Where cases come from

Production logs are the best source when they exist, because they carry the real distribution: the typos, the truncated PDFs, the unusual formats and the users who do three things in one message. Support tickets, sales transcripts, code review comments, incident reports and document repositories are also useful. Include typical, edge and adversarial cases, as OpenAI's evaluation guidance recommends (OpenAI, evaluation best practices). Each source has a bias:

• Production logs over-represent whatever the current system handles and under-represent requests it refuses or never receives.
• Support tickets over-represent failures, because happy users do not write in.
• Expert-authored cases over-represent what experts find interesting and under-represent mundane volume.
• Synthetic cases over-represent the generator's imagination and can encode its blind spots.

Use at least two sources and label the provenance of every task. If production traffic is unavailable, say so in the report.

Use authorised access and minimised or synthetic copies. Redact credentials and unnecessary personal or confidential data; keep restricted source material outside general-purpose repositories, with access and retention controls. Preserve provenance and disclose any resulting evidence gaps.

The sampling frame and stratification

Define the population first. "All supplier invoices received in the last six months" is a population. "Invoices from the finance shared drive" is a convenience sample unless you can show it is representative.

Then stratify. Useful dimensions include:

• Intent or task type: extract, classify, summarise, answer, act.
• Frequency: common, occasional, rare.
• Difficulty: routine, ambiguous, adversarial.
• Risk: low, medium, high consequence.
• Format or modality: text, table, scanned image, long document.

Over-sample rare but costly cases, because a simple random sample may contain none of them. Record the true prevalence of each stratum so you can compute a production-weighted score later. If 2% of traffic is a high-risk case, a set that is 25% high-risk gives you useful information about that stratum but an unweighted average that overstates its importance.

Anatomy of a task

A task record should contain everything needed to run and score the case without a meeting. A workable schema:

id: inv-2026-04-1187
input: "Path or inline text of the invoice"
context:
  - "Supplier master record for ACME Pty Ltd"
  - "Purchase order PO-4471"
constraints:
  - "Return JSON matching invoice.schema.json"
  - "Do not infer line items that are not printed"
expected:
  oracle: "reference.json"
  match: "field_exact"
metadata:
  source: "production_log"
  collected_at: "2026-04-18"
  stratum: "scanned_invoice_high_value"
  difficulty: "medium"
  risk: "high"
  prevalence: 0.02
  holdout: true

The schema forces three decisions: what context is available at inference time, what constraints the system must respect, and what the oracle is. A task without an oracle is a prompt, not an evaluation item.

Writing the oracle

Prefer an executable oracle:

• an exact or normalised answer;
• a passing test suite;
• a correct database state;
• a valid schema plus field-level checks;
• a reference answer with an explicit rubric.

For open-ended outputs, write the rubric before you collect many cases. If you cannot state what a good answer looks like, you are not ready to scale the evaluation. That is a useful finding, not a delay. Rubric design is covered in objective metrics and rubrics.

Labelling without importing your own bias

Labels are data too. A sloppy labelling process produces a test set that measures the labeller's habits rather than the system's behaviour.

A practical protocol:

1. Write the labelling guide first. Define each category, the decision boundary and at least one example per boundary.
2. Pilot on 20 cases. Have two people label them independently. Discuss every disagreement. Revise the guide.
3. Double-label a subset. 10–20% is a reasonable start. Keep the labels independent until both are recorded.
4. Adjudicate disagreements. Record the final label and the reason. Disagreements are the most informative cases in the set.
5. Measure agreement. Report it. If agreement is low, fix the guide before collecting more data.
6. Audit for drift. Relabel a sample after a few weeks; people's interpretations change.

Do not let the person who wrote the prompt also label the outputs without a blind. They will recognise their own work and grade it generously. Criteria can also drift as reviewers see outputs, so build in a revision round (Shankar et al., 2024).

How many tasks?

The right sample depends on the effect you need to detect, the variance of task scores and the correlation between candidates.

A pilot can expose gross failures and estimate variance, but no fixed task count is automatically decision-grade. For independent binary outcomes, a rough precision plan is n ≈ 1.96² × p × (1 − p) / margin²; at the conservative p=0.5, a ±5-point margin gives about 385 independent cases. This is a planning approximation, not a guarantee for rare events, paired differences, clustered samples or repeated selection.

For a model comparison, plan from the paired-difference variance and minimum meaningful effect, with the desired power and error rate. Document groups such as documents or conversations, and account for their within-group dependence. Miller (2024), Card et al. (2020) and the uncertainty article explain the distinction. A pilot’s value is what it reveals and what it lets you plan, rather than a universal minimum size.

Splits and versioning

Keep three sets:

• Development set: used to iterate on prompts, tools and scaffolding.
• Sealed holdout: used only for the decision run. No prompt tuning, no threshold fitting, no repeated peeking.
• Fresh set: collected after a model update or a process change, to check whether the result still holds.

Version every change to the dataset. A score is meaningless without the dataset version, because adding ten hard cases can move the average more than a model upgrade. Version-control non-sensitive task definitions. Keep restricted raw production data and holdouts in appropriately controlled storage, with minimisation, access and retention rules. See contamination, leakage and holdouts for the access controls that keep a holdout sealed.

Synthetic data: useful, with caveats

Synthetic cases are good for covering rare formats, testing schema edge cases and generating adversarial inputs. They are poor at representing prevalence: a generator produces what you ask for, not what your users do.

If you use synthetic data:

• label every record as synthetic;
• keep it in a separate stratum;
• never let it dominate the weighted score;
• check it against a sample of real cases;
• disclose whether the evaluated model also generated cases; validate them independently and assess generator-related blind spots before drawing a generalisation claim.

A synthetic set can double your coverage in a day. It can also double your confidence in a distribution you invented.

Failure cases to avoid

The convenience sample. Twenty examples from one team's inbox. It is a demo.

The ambiguous label. Two experts would grade it differently and the guide does not say which is right. Fix the guide or drop the case.

The leaked answer. The expected output appears in the context, the file name or the system prompt. The model looks brilliant and has learned nothing.

The multi-skill task. One case tests extraction, classification and tone simultaneously, so a failure does not tell you what to fix. Split it.

The happy-path set. Only clean PDFs, polite users and well-formed requests. Production is not like that.

The set that became a dev set. The holdout was used for prompt tuning until it stopped being a holdout. Rotate it.

A minimum dataset review

Before you run a model, review the set against five questions. Benchmark quality depends on lifecycle practices, not just task count (Reuel et al., 2024).

1. Does each task have a clear oracle or rubric?
2. Is every task traceable to a source and a date?
3. Does the stratum mix match the decision, including rare high-risk cases?
4. Has a second person labelled a subset, and is the agreement reported?
5. Is there a sealed set that no one has tuned against?

If any answer is no, fix it before collecting more cases. A smaller, well-labelled set beats a larger, ambiguous one.

The point

An evaluation dataset is an argument about what matters: its sampling frame says which work you represent, its oracle says what counts as success, and its labels say whose judgement you trust. Make those choices explicit. The next step is to protect it from leakage, which is covered in contamination, leakage and holdouts.

References

• Card, D., Henderson, P., Khandelwal, U., Jia, R., Mahowald, K. and Jurafsky, D. (2020). With Little Power Comes Great Responsibility. EMNLP 2020. Read source
• Miller, E. (2024). Adding Error Bars to Evals: A Statistical Approach to Language Model Evaluations. Read source
• OpenAI (2026). Evaluation best practices. Read source
• Reuel, A., Hardy, A., Smith, C., Lamparth, M., Hardy, M. and Kochenderfer, M. J. (2024). BetterBench: Assessing AI Benchmarks, Uncovering Issues, and Establishing Best Practices. NeurIPS 2024. Read source
• Shankar, S., Zamfirescu-Pereira, J. D., Hartmann, B., Parameswaran, A. G. and Arawjo, I. (2024). Who Validates the Validators? Aligning LLM-Assisted Evaluation of LLM Outputs with Human Preferences. Read source