Keep your AI benchmark honest: datasets, holdouts and leakage

A model can appear to ace a test because it learned the underlying skill, because it has seen similar material, or because the evaluation accidentally gave it the answer. These explanations matter differently. A useful benchmark needs data that represent the task, a split that tests the intended kind of generalisation, and controls that limit information flowing from evaluation into development.

This guide is about practical dataset integrity. It complements the task-design article and the broader model benchmarking guide.

Name the kind of generalisation you need

A holdout is only meaningful relative to a question. Do you want to know whether a model can handle new examples drawn from the same workflow, future cases after a policy change, unfamiliar users, new document templates, or a task never used during prompt development? These require different splits.

A random split may be appropriate when examples are independent samples from a stable population. It can be misleading when near-duplicates, repeated customers or shared source documents appear on both sides. Group records by the unit that could leak: customer, incident, conversation, source document, repository or template. A chronological split can test future performance, but a later period may also differ in prevalence or policy, so describe what changed.

For an assistant that answers questions over internal policy documents, make clear whether the test checks knowledge of fixed documents, retrieval over new documents, or general reasoning about policy. Pin document versions and retrieval configuration. A changing corpus can make yesterday’s reference answer incorrect today. Track the task, corpus and expected answer versions together.

Understand different forms of leakage

Leakage is information unavailable in the intended deployment that makes evaluation easier. It may come from accidental train-test overlap, duplicated examples, a label or answer hidden in metadata, prompt instructions that expose the gold answer, a tool that can query the answer key, or benchmark items included in public training data. The remedy depends on the path.

Use separate storage and permissions for holdout inputs and labels. The evaluation runner should retrieve labels only for scoring, not include them in model context. Inspect test templates for answer-bearing phrases and ensure retrieval indexes exclude the answer file. Review logs for accidental prompt inclusion. For internal data, keep a provenance field so you know whether an item was authored, adapted from a public benchmark or derived from customer interaction.

Public benchmark contamination is difficult to prove from a model’s fluent response alone. Training data may be undocumented, and exact-string searches miss paraphrases or partial exposure. Yang et al. study training on paraphrased or translated benchmark samples, showing how this contamination can evade overlap checks; they also evaluate a model-based decontamination method (Rethinking Benchmark and Contamination for Language Models with Rephrased Samples, version 2). This is evidence about contamination and detection, rather than an original-versus-rewritten inference test that proves memorisation. A newer 2026 preprint argues that contamination may inflate absolute scores without necessarily changing the relative order of a large leaderboard in its studied benchmarks; its result is specific to its data and method, not a reason to ignore leakage risk (Xiao and Cheng, 2026). Report what you checked and what remains unknown.

Treat holdouts as a finite resource

A holdout loses value when its answers influence development. Looking at a failure is useful for diagnosis, but if you then alter the prompt to fix it, that item has become training feedback in the broad evaluation sense. This does not mean you must never inspect errors. It means you should label the set’s role honestly and maintain a fresh final set for a decision that needs independent evidence.

A practical arrangement has three partitions:

• Development: accessible for prompt, rubric and code iteration.
• Validation: used for candidate selection or threshold tuning, with logged access.
• Final holdout: accessed only for a predeclared comparison or release gate.

Small teams may not have enough data for three large splits. They can use a compact final set, a time-based refresh, or leave-one-group-out comparisons, provided the limitation is clear. The name of the split matters less than the discipline around how it is used.

Version changes instead of erasing them

Keep each benchmark release immutable. Give it an ID, date, task statement, inclusion criteria, data hash or version, label guide, split assignment and known limitations. If a label is corrected, record an erratum and create a new version. If you add incidents, do not silently merge them into the past score series.

When scores must be compared across versions, run both versions for a period and report the bridge. A new set may be harder because it contains more edge cases; a score drop could reflect a real regression, a changed sample or both. Maintain an anchor subset of stable cases for continuity, but do not let a public anchor become the only target the system learns to game.

Protect data that should not be public. If results are shared externally, release task descriptions, aggregate counts and scoring logic where possible without exposing personal or confidential inputs. A benchmark can be transparent about methods while restricting raw data. Document such restrictions so readers know what independent replication can and cannot cover.

A contamination review you can run

Before a final comparison, walk through these checks:

1. Search for exact and near-duplicate items across development, validation and holdout.
2. Group examples by source or entity and verify the split unit matches the generalisation claim.
3. Inspect prompts, retrieval indexes, metadata and logs for hidden answers.
4. Confirm the runner cannot read answer keys before it records the model output.
5. Record whether items are public, synthetic, adapted or private, and what training exposure is known.
6. If a public benchmark is central, consider paraphrase or fresh-item checks and state their limits.
7. Review holdout access history and move repeatedly inspected items into development.
8. Pin data, labels, policy documents and scoring code by version.

No checklist can establish that a closed model never encountered public material. The honest conclusion may be “training exposure could not be verified”. That is still useful information if readers can see how the test was designed to reduce avoidable leakage.

Fresh examples are not automatically better

Private cases reduce the risk that benchmark answers are publicly indexed, but privacy alone does not make a set representative. A private convenience sample may overrepresent one customer, product version or failure type. Synthetic cases can fill coverage gaps, yet a model may react differently to generated text than to naturally occurring inputs. Use several sources, compare their distributions, and show results by source when the distinction matters.

Freshness also has a cost. New cases may have less reliable labels or shorter observation windows. For a support workflow, an unresolved case may not have a settled correct answer until a specialist investigates it. Mark uncertain labels, exclude them from strict accuracy claims or score them under an explicit rubric. Do not force a crisp ground truth where the underlying decision is genuinely ambiguous.

The goal is not a perfectly uncontaminated dataset; that may be impossible to demonstrate. It is a test whose provenance and limitations are legible, whose split matches the question, and whose score cannot be improved simply by reading the answer key. Pair it with a transparent scoring plan, repeat runs when output variability matters using the guide to repeatability, and refresh it with the process in evaluation maintenance. For public tests, see the checklist for reading benchmarks.