Interpret AI evaluation results and keep them useful over time

An evaluation is complete when it informs a decision, not when a dashboard turns green. A useful report shows the conditions, outcomes, uncertainty and failures that explain what the result can support. It also makes clear when the evidence is too weak to decide.

Imagine a fictional team comparing two configurations of a policy-answering assistant. Both have been run on the same protected cases. One is faster; the other has fewer unsupported claims in the observed sample. These are illustrative possibilities, not benchmark results. The right choice depends on the predeclared quality gates, the uncertainty and the consequences of errors, not on a single blended rank.

Read the scorecard in layers

First check whether the run is valid: did both candidates receive the intended inputs and tools, were configurations frozen, and were errors and exclusions handled consistently? A score from a broken harness or mismatched dataset is not a reliable comparison. Preserve the run record, fix the problem and apply a consistent rerun policy rather than quietly replacing inconvenient results.

Next examine critical failures and minimum requirements. Count unsupported claims, unauthorised actions, data exposure or other locally defined serious failures separately. Read representative examples and assess whether the rubric and reference were correct. An average can conceal a failure concentrated in a small but important slice.

Then compare ordinary quality by slice, paired difference and uncertainty. A candidate may be better on straightforward cases but worse on conflicting policies. Report case counts, denominators and intervals or other uncertainty measures where appropriate. The uncertainty guide explains why a small observed lead is not automatically a reliable improvement. If the data cannot distinguish a meaningful improvement from a meaningful regression, say the comparison is inconclusive.

Finally, compare latency and cost for candidates that satisfy the quality gates. Report the workload, timing conditions, token usage, retries and relevant price assumptions. Latency and cost shows why cost per attempt and cost per accepted task answer different questions. If a human must review every answer, include that operational burden in the decision.

Write a decision record

A concise record should identify the decision owner, date, candidate and baseline versions, case and rubric versions, comparison rules and intended population. Include overall and slice results, critical failures, missing outcomes, uncertainty, latency, cost and notable limitations. Link to the run manifest and supporting evidence under the right access controls.

State the decision in operational terms: for example, retain the baseline, run a limited pilot with human review, collect more cases, or adopt a candidate for a defined subset. Specify what would trigger rollback or escalation and who will monitor it. Do not infer that a candidate is safe for a different domain or authority level simply because it passed the current test.

A result can be statistically inconclusive but operationally useful. If both candidates meet a minimum and one is substantially cheaper, the team may choose the cheaper option while acknowledging uncertainty about quality. If a critical failure is observed, the team may narrow scope or add a control even if the overall score improves. These are decision rules, not universal statistical conclusions.

Turn production learning into tests

With appropriate authorisation, privacy controls and retention, review real failures, user feedback and changing task distributions. Capture a minimal reproducible case, source version, expected behaviour and reason it matters. Add it to the development set first. Ask a domain reviewer to verify the label and check that sensitive information is removed or protected.

Group recurring failures into root causes: missing retrieval, incorrect interpretation, unsupported claim, tool error or ambiguous policy. A single incident can create several related cases, but keep their shared origin visible so they are not mistaken for independent evidence. Fix the system, then run targeted regression cases and the broader suite. An improvement on the new case does not prove that unrelated behaviour stayed intact.

The OpenAI evaluation best practices guide recommends continuous evaluation and feedback-calibrated scoring. The practical cadence should reflect the risk and rate of change: a high-impact workflow may need checks on every relevant change, while a low-volume internal experiment may use a scheduled review. Define the cadence and owner explicitly.

Version the measurement, not just the model

Track the dataset, partition manifest, reference answers, rubric, judge, harness, prompt, retrieval corpus, tool versions and model identifier. A change to any of these can change the result. If you revise a rubric, do not compare the new score directly with an old score without explaining the change. Where feasible, rescore a common set under both versions to understand the discontinuity.

Keep a stable baseline suite for trend analysis and a rotating protected set for fresh evidence. The stable suite becomes familiar over time; the fresh set may differ in difficulty. During a transition, run the baseline and candidate on both sets and report each result separately. The holdout guide explains why repeated inspection erodes independence.

Retain old run manifests and outputs where policy permits. Correct erroneous labels transparently and record which historical results are affected. If a provider changes a model alias or pricing, note the date and verify the current configuration before rerunning. A score trend across different systems, datasets and rubrics is not a clean model trend.

Set a review trigger

Rerun relevant tests after changes to the model, prompt, retrieval index, tool, output parser, rubric or business policy. Also review when user populations, task mix or observed failure patterns shift. A dashboard can flag a change, but a responsible person needs authority and time to investigate it.

Close the loop with a short post-decision review: did the pilot produce the expected benefit, what failed, which cases were added, and does the deployment scope remain justified? Keep the original decision and evidence rather than rewriting history to match the outcome. The cluster overview connects this maintenance loop to the initial benchmark and system-selection decision.

Make a change review reproducible

When a provider releases a new model version, first capture the old and new identifiers and the proposed configuration. Run a small development smoke test to catch formatting or tool compatibility issues. Freeze any necessary adaptations, then compare the candidate and baseline on the same protected cases and scoring version. Review critical failures and operational measures before changing the routing rule. If the old version is no longer available, state that the comparison relies on historical baseline evidence and may be affected by changes in environment or workload.

After a decision, archive the manifest and record the owner, rollout scope and next review date. If the system is rolled back, preserve both the trigger and the evidence that led to the rollback. This history is useful when a later candidate appears to reproduce an earlier failure. Do not treat a failed experiment as wasted work: a well-labelled regression case and a clear account of the failed condition can prevent the same problem from recurring. Retest the control after changes to the surrounding system, because an old fix can be invalidated by a new tool or policy.

A completed decision record

This is a filled teaching record, not an actual approval or measured pilot. It demonstrates an operational decision that preserves uncertainty instead of declaring a universal winner.

Return to the Terra cluster guide · Inspect the four-way results and evidence.