Keep AI evaluations useful as models and products change

An evaluation is not a one-off launch gate. Models, prompts, data, tools, policies and user behaviour all change. A score from six months ago may refer to a different system and a different task. Keep the evaluation useful by versioning it, refreshing cases deliberately and preserving enough history to interpret movement.

This article turns the cluster’s benchmark design into an operating routine.

Version the whole experiment

Give each evaluation release a version that identifies its task definition, dataset, splits, label guide, scoring code, prompts, tools and environment. Store the model and configuration used for each run separately from the benchmark version. This helps distinguish a model change from a test change. Include release dates and a short changelog that explains additions, corrections and removals.

Keep raw outputs and item-level scores where policy permits. Preserve the analysis script or notebook and the thresholds used for decisions. If an item’s expected answer changes because a policy was updated, keep the old label with its context and issue a new version. Silent edits make score histories impossible to trust.

Build a continuous evaluation loop

Run a small, fast suite on prompt, retrieval, parser and tool changes. Run a broader suite before releases and after model changes. Review production incidents and user feedback for new cases, then add verified examples to development data. OpenAI’s evaluation guidance recommends continuous evaluation on changes and growing a suite based on observed nondeterminism and failures (evaluation best practices). Adapt the cadence to your risk and cost; not every system needs a large run on every commit.

A simple pipeline can include:

• a smoke set for schema, safety and critical paths;
• a regression set of known failures and ordinary cases;
• a protected holdout for release decisions;
• a monitoring sample from recent real use, reviewed under privacy controls.

Each serves a different purpose. The smoke set catches obvious breaks quickly. The regression set ensures fixes remain fixed. The holdout estimates broader performance. The monitoring sample detects drift and new failure modes. Do not merge their results without identifying the source.

Refresh without breaking comparisons

New examples may make a suite more representative but also harder. Keep a stable anchor set for historical comparison and a rotating recent set for current conditions. Report anchor and new-set scores separately. If you replace the entire dataset, run old and new versions in parallel on at least one system to estimate how the change affected results.

Retire examples that are invalid, duplicated or no longer relevant, but preserve a record of the reason and the old score. An item can also be kept as a historical regression test even if it no longer reflects current traffic. Mark its role. Avoid letting every known test case remain in the primary quality score forever; the suite can become a memorisation target and stop representing the workload.

Watch for distribution and grader drift

A stable score does not mean the system is stable if the task mix has changed. Monitor input categories, language, user groups, document formats, retrieval sources and escalation rates. Compare recent samples with the distribution used for the benchmark. If one slice grows or a new policy changes the task, update the sample and rubric.

Automated graders drift too. When a judge model or prompt changes, rerun a calibration set with human labels. Test score stability on known examples and inspect changed judgements. If humans revise the rubric, version it and consider rescoring historical outputs only as a clearly labelled retrospective analysis. Do not compare scores from two grader versions as though the scale were identical without checking.

Set decision thresholds and owners

Define who owns the evaluation, who can approve dataset changes, and who can override a release gate. A threshold should correspond to a risk or service target, with a clear fallback when the result is uncertain. Require a documented reason for exceptions and track any follow-up. Thresholds are not sacred: adjust them when product risk or user expectations change, but explain the change and preserve prior decisions.

Use trend charts as signals, not automatic proof. A sudden score drop may reflect a broken prompt, a provider change, a changed sample or a scoring bug. Investigate item-level differences before rolling back. An unexpected improvement can also indicate leakage or grader failure. Pair dashboards with examples and a named escalation path.

Example release routine

A hypothetical team updating a policy-answer assistant might run 40 smoke cases on every prompt change, 300 regression cases nightly, and a protected 200-case holdout before selecting a new model. Once a month, reviewers sample recent user questions and add confirmed failures to the development set. Every quarter, they inspect whether policy documents or question patterns changed. These counts and intervals are illustrative, not universal advice.

The team pins policy snapshots, prompts, model identifiers and grader versions. If it changes the policy corpus, it creates a new evaluation release and runs a bridge comparison. A release note says whether the system improved on old anchor cases, new cases or both. If the judge changes, human-labelled calibration items test whether the score meaning moved.

Protect privacy while learning from production

Production logs can surface realistic cases, but they may contain personal information, confidential business data or credentials. Apply data minimisation, access control, retention limits and redaction review. Preserve the behaviours needed for evaluation while removing unnecessary identifiers. If redaction changes meaning, mark the item or create a controlled synthetic case instead.

Do not assume that internal access automatically makes reuse appropriate. Follow the organisation’s data policies and applicable obligations. Keep a record of the source and transformation so reviewers know what the benchmark represents. When raw cases cannot be shared, publish the process, counts and limitations rather than implying that results can be independently reproduced from inaccessible data.

A maintenance checklist

At each significant release, confirm: benchmark and configuration versions are pinned; new cases have verified labels; changes are explained; the holdout remains protected; graders are calibrated; distributions and critical slices are reviewed; cost and latency assumptions are current; and unresolved failures have an owner. Keep a dated report with results, uncertainty and the release decision.

The objective is continuity without pretending the workload is static. A frozen suite gives historical comparability but can become stale; a constantly changing suite reflects current use but can obscure trends. A stable anchor, a rotating current sample and explicit version bridges balance those needs. Use the detailed guidance on holdouts and leakage, human and model-judge calibration, and public benchmark interpretation when updating the suite. When a score moves, you should be able to tell whether the product changed, the task changed or the measuring instrument changed.