Build holdouts that make AI evaluations more trustworthy

A holdout is a collection of cases withheld while a system is developed, then used to assess a frozen candidate. Its value is that it tests performance beyond the examples used for decisions about prompts, tools and scoring. It is not magic: a leaked, unrepresentative or repeatedly consulted holdout can give a misleading result.

For a fictional policy-answering assistant, suppose the team collects questions about expenses, leave and purchasing. If ten paraphrases of the same purchasing policy are split randomly across development and holdout, the apparent test set contains familiar evidence. A better split keeps related documents and their derived questions together. These examples are illustrative, not measured findings.

Know what can leak

Leakage is information crossing a boundary that the evaluation assumes is protected. The obvious case is copying test questions into prompt development. Less obvious examples include a reference answer accidentally included in a tool response, a near-duplicate document in both partitions, or adjusting the rubric after seeing which candidate it favours. Repeatedly checking a holdout and tuning to its failures also turns that holdout into development feedback, even if nobody copies the file.

Training-data contamination is a related but distinct concern: benchmark items may have been encountered during model training or later adaptation. With closed training data, you may not be able to determine whether a given item was seen. Balloccu and colleagues describe direct and indirect exposure routes and reproducibility problems in benchmark use. Their analysis does not prove that any particular response is memorised. Keep that uncertainty separate from leakage under your own team's control.

Choose the split unit before you split

Split on the source of dependence, not blindly on rows. Related questions from the same document, customer, conversation or incident should normally stay in one partition. If the intended use is on future policy revisions, a time-based holdout may better represent the decision: develop on earlier versions and test on later ones, taking care that later answers were not available during tuning. If the intended use is across new customers, hold out customer groups where permissions and sample size allow.

There is no universally correct split. Grouping can leave very few independent groups, making estimates unstable. A time split may also combine genuine future difficulty with unrelated changes in policy or traffic. Document the target population and the reasons for the split so a reader can judge the trade-off. Keep a manifest of case IDs, group IDs, source versions and partition assignments; do not rely on a filename alone.

A practical arrangement includes development, validation and a final holdout. Use development to explore cases and improve prompts. Use validation for decisions during iteration, recognising it too becomes familiar over repeated cycles. Restrict the final holdout and set a planned evaluation cadence. Small teams may combine development and validation but should still keep a separate final set if the decision warrants it.

Protect the boundary in practice

Limit access to raw holdout cases and labels to the people or service responsible for evaluation. Store a versioned manifest and log each use. The person tuning prompts can receive aggregate results when appropriate, while detailed failure examples remain restricted until the decision is complete. This is an operational control, not a guarantee: even aggregate feedback can be exploited if checked often enough.

Freeze candidate settings and scoring before the final run. Record who approved the freeze, the case version, the run date and any deviations. Predefine what happens to provider errors or infrastructure failures. If a bug in the evaluation is discovered, correct it transparently, preserve the original run and rerun all candidates under the same rules. Do not discard a case merely because its result is inconvenient.

Check near duplicates across partitions using document IDs, hashes and similarity search, followed by human review. Exact matching catches identical text but misses paraphrases; semantic similarity can flag unrelated cases and cannot prove independence. Consider shared source material, templates and answer logic as well as surface wording. Record residual uncertainty, particularly where a model's training corpus is unavailable.

Keep the data usable and authorised

Every case should have provenance: source, creation date, policy or document version, permission basis, transformation and reviewer. Remove unnecessary personal information and restrict access to sensitive cases. Redaction can alter the task, for example by removing the distinction a reviewer needs to judge an answer. Test that the redacted example remains meaningful and mark any limitations.

Keep labels separate from model inputs. A reference answer may be provided to a grader, but it must not accidentally appear in the system's prompt unless that is part of the intended task. Store references and rubrics in version control or another auditable store with appropriate access restrictions. When a policy changes, a previously correct label can become wrong; bind each case to the source version it represents.

Synthetic data can fill missing scenarios, but identify it as synthetic and independently review both the prompt and expected outcome. If the same model family generates and judges examples, correlated errors may make the evaluation look stronger than it is. Use a mixture of sources where possible, and check whether generated language resembles real users' requests.

Refresh without losing the baseline

Retire or rotate a holdout when it has been repeatedly inspected, exposed, or no longer represents the workload. Preserve the old version and its history so results remain interpretable. A fresh set can be harder or easier than the old one; a score change across two different sets is not necessarily a system change. Where feasible, run a frozen baseline and candidate on both sets during a transition and report the overlap and limitations.

A holdout does not replace slice analysis or human review. It only provides cleaner evidence about the cases it contains. For deciding which cases belong there, use task design; for translating results into a cautious decision, use repeatability and uncertainty.

Respond to suspected exposure

If someone discovers that a holdout case appeared in a prompt example or a tuning discussion, record the affected IDs, when the exposure occurred and which configurations may have benefited. Do not quietly delete the case after seeing its outcome. Assess whether related cases from the same source also need to be treated as exposed. Report results with and without the affected group only if the analysis is labelled as a sensitivity check, not as a new unplanned way to select a winner.

For a consequential decision, create a fresh protected set before drawing a strong conclusion. A fresh set still needs representative coverage and independently checked labels; secrecy alone does not make it useful. Preserve the exposed set as a regression resource and document its new role. If fresh data is unavailable, a limited pilot with additional human oversight may be an option, but the decision record should explicitly say that independent holdout evidence is missing. Assign an owner and a date to revisit that limitation.

A holdout exposure record

The table is a hypothetical incident record. It shows why removing a row after seeing its outcome is different from documenting exposure and renewing the test.

Return to the Terra cluster guide · Inspect the four-way results and evidence.