Interpreting Results and Keeping Evaluations Useful Over Time

A results table arrives with a recommendation attached. Model B is "better". The number is higher, the chart is green, and the meeting is in twenty minutes. The useful question is not whether the number is higher. It is what the number justifies, under which conditions, for how long.

This final article is about reading a result and keeping the evaluation alive, completing the loop in the pillar guide. It gives a checklist for interpretation, a one-page decision memo, and a maintenance plan for the things that rot: model versions, prompts, judges, prices and holdouts.

Read the claim, not the rank

A benchmark result is a claim with several parts: a question, a population, a sample, a metric, a baseline, a configuration and a date. If any part is missing, the claim is weaker than it looks.

Ask:

1. What question was the evaluation designed to answer? If it was designed to compare general capability, it cannot answer a question about your invoice workflow.
2. What population was sampled? Production traffic, expert-authored cases or synthetic data? Each has a different bias.
3. How large was the sample, and how many trials? A 100-task single-run comparison may be only a pilot; assess adequacy against the decision, dependence and uncertainty.
4. What exactly was measured? Accuracy, pass^k, cost per success, latency? A composite score can hide the dimension that matters.
5. What is the baseline? The current system, a previous model, or a straw man?
6. What is the uncertainty? If there is no interval, you do not know how much of the difference is noise (repeatability and uncertainty).
7. What configuration produced it? Model snapshot, prompt, tools, reasoning effort, retries, region.
8. Who ran it, and what did they gain from the result? A vendor comparison is evidence, but it is not independent evidence.

These questions are not cynicism. They are how you decide how much weight to put on the result.

Statistical versus practical significance

A difference can be statistically distinguishable from noise and still be irrelevant. A one-point gain that costs 40% more per task may not be worth shipping. A five-point gain with a wide interval may be worth investigating further if the error cost is asymmetric.

Tie the result to the decision threshold. If the threshold is "the lower bound of the interval must exceed 90% on the high-risk stratum", a point estimate of 91% with a lower bound of 84% does not meet it. If the threshold is "no more than a 2% policy-breach rate", a difference in average tone is beside the point.

A significance test assesses evidence against a specified null under its assumptions; it does not prove an effect exists or matters. Report effect size and uncertainty alongside the practical decision threshold.

Multiple comparisons and cherry-picking

If you test five models on six benchmarks with three prompt variants, you have 90 configurations to inspect; the number of statistical comparisons depends on the contrasts you test. The best of them will look impressive even if all the models are equivalent. This is the multiple-comparisons problem, and it is endemic in leaderboard culture. It is also a reporting problem: most benchmarks do not report statistical significance or allow easy replication (Reuel et al., 2024).

Practical responses:

• pre-register the primary comparison and the decision threshold;
• report all the comparisons you ran, not just the winner;
• replicate the finding on a fresh sample before committing;
• treat subgroup discoveries as hypotheses, not conclusions;
• adjust for multiple testing when the decision depends on many comparisons.

A result that appears only after selecting the best model, benchmark and prompt is a candidate for replication, not a finding.

External validity

An evaluation is local. It is valid for the tasks, users, tools and time it sampled. Change the user population, the document formats, the tool versions or the model snapshot, and the result may not hold.

Ask what would have to change for the result to flip:

• a new document format;
• a different user cohort;
• a tool API change;
• a model update;
• a price change;
• a policy change;
• a shift in traffic mix.

If the answer is "almost anything", the result is fragile. That is useful to know before you build a process around it.

Goodhart's law

When a measure becomes a target, it stops being a good measure. The evaluation version of this is benchmark overfitting: prompts, scaffolds and model choices are tuned against the test until the score improves without a corresponding improvement in the world. The Leaderboard Illusion analysis shows how selection effects and unequal access can distort a public ranking (Singh et al., 2025); the same dynamics operate inside a company.

The controls are structural: a sealed holdout, a fresh set, pre-registered decisions, and a culture that rewards finding failures rather than maximising a number.

The decision memo

A one-page memo is enough:

Decision: [what we are choosing]
Options: [A, B, status quo]
Criteria: [quality, cost, latency, risk, constraints]
Evidence: [task set, n, trials, metric, interval]
Result: [mean and interval per option; paired difference]
Cost: [cost per successful task; p95 latency]
Failure mix: [top three failure categories per option]
Limitations: [what was not tested; residual contamination risk]
Recommendation: [option and conditions]
Owner and review date: [name, date]

The memo forces the team to state the decision, the evidence and the conditions. It also creates a record for the next person, which is the difference between an evaluation and an anecdote.

Maintaining the evaluation

An evaluation is a living asset. It needs versioning, ownership and a cadence.

Version everything that affects the result: task set, prompt, tools, harness, judge, model snapshot, pricing date. A score without versions cannot be compared with a later score.

Schedule re-runs. A reasonable cadence is weekly or per release for a regression suite, monthly for a broader evaluation, and on every model or prompt change. The right cadence depends on how fast the system changes.

Monitor production. Log failures, user corrections, escalations and refusals. Convert incidents into test cases. The production distribution is the best source of new evaluation tasks, and the failures the test set missed are the most valuable additions.

Refresh the holdout. Rotate a portion of the set on a schedule, and retire a set after it has been used for a decision (contamination, leakage and holdouts). Keep an anchor set for comparability.

Re-validate the judge. A judge model can be updated or deprecated. Keep a frozen calibration set and re-check agreement after any change (human evaluation and model judges).

Watch deprecation. Models are retired, APIs change, prices move and vendor platforms are replaced. The OpenAI Evals platform, for example, is scheduled to become read-only for existing users on 31 October 2026 and to shut down on 30 November 2026 (OpenAI, evaluation best practices). If your evaluation lives only in a vendor dashboard, that is a dependency risk. Keep the task set, the harness and the results in a form you control.

Refresh selection tasks while retaining critical regression gates. A saturated selection benchmark may stop distinguishing candidates, but a universally passed policy or safety gate can still catch future regressions. Keep those gates; add harder or more relevant selection tasks and record the change.

A maturity model

Level 1: ad hoc. Demos, intuition and vendor claims. No repeatable task set.

Level 2: regression suite. A versioned task set, a scorer and a run on every change. Good for preventing regressions, weak for choosing between close candidates.

Level 3: continuous evaluation. Production sampling, judge calibration, uncertainty reporting, cost and latency on the same dashboard. Good for operating a system.

Level 4: decision system. Pre-registered decisions, sealed holdouts, periodic re-baselining, incident-to-test feedback and a documented owner. Good for making repeated bets with evidence.

Most teams do not need Level 4 on day one. They do need to know which level they are at, and what the next step is. A team at Level 1 that builds a 100-task regression suite has made a larger improvement than a team at Level 3 that adds another benchmark.

Failure cases

The dashboard nobody reads. Metrics are collected but not connected to a decision. Delete the dashboard or give it an owner and a review.

The holdout that became a dev set. The set was used for tuning until it stopped being independent. Rotate it and record the exposure.

The judge that was never re-validated. The judge model changed; the agreement numbers are stale.

The evaluation that outlived its decision. The product pivoted, but the test set still measures the old task.

The score that became a target. The team optimises the metric and stops looking at the failure cases.

The point

Interpreting a result is an act of judgement, not arithmetic. The number is necessary but not sufficient. State the claim, quantify the uncertainty, tie it to a decision, and write down the conditions. Then keep the instrument in good repair. That is how an evaluation stays useful after the launch meeting.

References

• OpenAI (2026). Evaluation best practices. Read source
• Reuel, A., Hardy, A., Smith, C., Lamparth, M., Hardy, M. and Kochenderfer, M. J. (2024). BetterBench: Assessing AI Benchmarks, Uncovering Issues, and Establishing Best Practices. NeurIPS 2024. Read source
• Singh, S., Nan, Y., Wang, A., D'Souza, D., Kapoor, S., Üstün, A., Koyejo, S., Deng, Y., Longpre, S., Smith, N. A., Ermis, B., Fadaee, M. and Hooker, S. (2025). The Leaderboard Illusion. Read source