Repeat AI evaluations and report uncertainty honestly

A model evaluation is a sample of behaviour under particular conditions, not a permanent property of a model. Outputs can vary between runs, cases differ in difficulty, and a hosted service can change over time. Repeating an experiment and reporting uncertainty makes a result more useful, but no interval can fix a dataset that does not represent the job.

Suppose two configurations answer the same fictional policy questions. One passes 82 of 100 cases and the other 78 of 100. Those invented numbers describe only that run on that set. They do not, by themselves, establish that the first configuration is reliably better on future work. The difference might be concentrated in a few cases, affected by repeated outputs or driven by the choice of cases.

Separate sources of variation

There are at least three useful questions. Case variation asks how outcomes differ across tasks drawn from a target population. Run variation asks how the same configuration behaves when a case is repeated. Environment variation includes provider changes, network conditions, tool state, load and grader changes. Record enough information to distinguish them rather than treating all variation as random noise.

A fixed seed, when supported, can aid reproduction, but it is not a promise that a hosted service or an entire agent environment is deterministic. Log model identifiers, timestamps, prompt and dataset versions, tool state, generation settings, retries, grader version and relevant service conditions. Preserve outputs or authorised references to them so a later reviewer can inspect the difference.

Repeat a representative subset of cases to estimate run-to-run variability, especially when a single failure has a high consequence. State how many attempts were made and whether the result is first attempt, best of several, majority vote or another rule. Best-of-many uses more resources and answers a different question from first-attempt performance. Apply the same policy to all candidates in the comparison or disclose the difference.

Compare paired cases

Run both candidates on the same case set. For each case, record whether A and B pass or fail and inspect the cases where they disagree. Pairing controls some case-difficulty differences: an easy case contributes to both candidates, while a case passed by one and failed by the other informs the observed difference. If outputs are repeated, keep the case identity and repetition nested within it.

A paired bootstrap can estimate uncertainty in the observed difference by resampling independent case units with replacement, calculating the score difference for each resample and summarising the resulting distribution. Resample groups instead of rows when cases share a document or customer, and preserve both candidates' results together. This procedure assumes the sampled units are a reasonable basis for the target population; it cannot correct selection bias or missing slices. With very few groups, the resulting interval may itself be unstable.

For a single pass proportion, a binomial interval can be useful when independent, similarly distributed trials are a reasonable approximation. NIST's discussion of exact binomial limits notes that simple normal approximations can be inaccurate for small samples or few failures. Choose and report the method, confidence level, numerator and denominator. A 95% confidence procedure refers to its long-run coverage under its assumptions, not a 95% probability that a fixed parameter lies in this particular frequentist interval.

Make the decision threshold explicit

Before running the final test, define a minimum improvement worth acting on and any non-negotiable failure limits. An interval spanning both a material improvement and a material regression is inconclusive for that decision. An interval that excludes zero can still include an improvement too small to justify migration. Conversely, failing to establish a difference does not prove equivalence. If equivalence is the goal, specify an acceptable margin and use an appropriate design.

In the invented 82-versus-78 example, inspect the paired outcomes and critical failures before applying a test. If both systems fail the same high-risk cases, choosing the higher aggregate score may be the wrong action. If there are only a few independent policy documents, 100 questions may overstate the amount of independent evidence. More repetitions of the same document do not create 100 independent documents.

Plan sample size around the decision and available budget. Rare critical failures require many relevant independent opportunities to estimate reliably; observing none in a small set does not demonstrate zero risk. If evidence is insufficient, collect more representative cases, narrow the claim or retain a human control. Report the limitation rather than manufacturing precision.

Report what the number covers

A useful result includes the dataset and split, number of independent units, repeats per case, score and interval method, paired difference, slices, missing outcomes and run dates. Show both individual candidate results and the comparison. Separate uncertainty due to sampling from known systematic limits such as a narrow language range, uncertain labels or a changing provider alias.

Do not pool runs across a model update without identifying the versions. A run before and after a change may answer a useful drift question, but it is not a single stable configuration. If an infrastructure outage affects one candidate, preserve the failure record and rerun under a consistent predeclared policy. The fair comparison guide covers configuration control, and maintaining evaluations explains how to track changes over time.

An honest interval is a statement about the evidence you collected and assumptions you made. It makes uncertainty visible so a team can choose whether to ship a limited pilot, gather more evidence or change the design. It is not a substitute for reviewing consequential failures.

Plan an escalation path for a close result

When a result is close to the decision boundary, first check the data and labels rather than repeatedly rerunning until a preferred answer appears. Review the paired disagreements, check whether a small number of source groups dominate the difference, and verify that missing outcomes were treated consistently. If the analysis was exploratory, label it as such; choosing a subgroup after seeing results can exaggerate apparent differences.

Then decide what additional evidence would be worth its cost. New independent cases from the target population can reduce sampling uncertainty, while repeats of existing cases help assess run variability. They address different questions. A fresh human review may resolve ambiguous labels but cannot make a narrow dataset representative. Record a stopping rule or decision date before collecting more data, especially if the evaluation will be checked repeatedly. If the required evidence cannot be collected, narrow the claim and use an operational control such as human review for the uncertain slice. Report the unresolved uncertainty in the handoff to the decision owner.

A worked paired example and a single-proportion interval

A passes 82 cases and B passes 78. The observed paired difference is (8 − 4) / 100 = 4 percentage points; the marginal totals alone would not reveal the twelve disagreements.

For A’s single pass proportion, a 95% Wilson interval with p = 0.82, n = 100, z = 1.96 uses centre (p + z²/(2n))/(1 + z²/n) and half-width z × sqrt(p(1−p)/n + z²/(4n²))/(1 + z²/n). This gives approximately 73.3% to 88.3%. NIST provides the Wilson formula. This is an interval for A’s proportion under the binomial assumptions, not an interval for A minus B and not proof of a meaningful difference. A paired analysis must retain the joint outcomes above. If the cases share policy documents, resample document groups rather than pretending all 100 rows are independent.

These invented outcomes are unrelated to the editorial benchmark scores. One cluster per producer and one AI judge do not support statistical intervals for the four-way quality bars.

Return to the Terra cluster guide · Inspect the four-way results and evidence.