Repeatability and uncertainty in AI model evaluations

One run gives you an observation, not a guarantee. Outputs may change across repeated calls, and a test set may contain too few examples to make a small score gap meaningful. A useful report distinguishes variability in model behaviour from uncertainty caused by which examples happened to be sampled.

This guide gives a practical design for repeated runs and uncertainty reporting. It assumes the task and scoring rules are already defined; see the benchmark pillar and metric guide first.

Name the sources of variation

Results can vary because of sampling settings, backend changes behind a model alias, nondeterministic serving, tool or retrieval changes, network failures, rate limits, timeouts or differences in test examples. A score can also vary because reviewers interpret a rubric differently. These sources are not interchangeable, so record them separately where possible.

A fixed seed may reduce variation in some systems, but it does not guarantee identical results across infrastructure changes or providers. If a service does not expose a stable version identifier or deterministic setting, say so. Store request parameters, timestamps, model names as returned, tool versions and raw outputs. Do not promise exact reproduction when the service makes it impossible.

Repeat the right thing

There are two simple experiments:

• Repeat the same cases with independent calls. This estimates output variability under the current configuration for those inputs.
• Sample new cases from the target population. This estimates how performance varies across the workload.

If both matter, combine them. For example, run each of 100 cases three times for two finalists. That creates 300 outputs per model, but there are still only 100 distinct cases. Do not treat all 300 as independent examples when calculating uncertainty over the task population.

Budget often limits repeats. Use a pilot to estimate how much variation exists, repeat a subset for expensive candidates, and prioritise cases where the decision is sensitive. For an application with high-stakes errors, repeat more and keep the evaluation set large enough to include relevant subgroups. A repeat schedule should be chosen before inspecting which candidate benefits.

Compare candidates on paired cases

Give every candidate the same cases under the same defined configuration. For each item, calculate the score difference between candidates. This paired design removes some noise from case difficulty: if a task is unusually hard for all models, it does not count as evidence for one model over another merely because one received easier examples.

For a binary outcome, inspect how often A wins, B wins and they tie on each paired item. For a numeric score, show the distribution of per-item differences, not only the difference between two averages. If repeated calls are available, account for the grouping of outputs by case. The correct method depends on the score and sampling design; do not choose a statistical test just because a library makes it convenient.

Report intervals with their assumptions

An interval communicates that an estimated score is uncertain. A bootstrap can resample independent evaluation units and recalculate a metric to estimate a range. The unit must match the design: resample conversations if conversations contain multiple related turns; resample repositories if tasks within each repository are correlated. Resampling individual turns as though independent can make an interval too narrow.

Research on benchmark variance discusses confidence intervals and tests that account for uncertainty in machine-learning benchmark scores (Bouthillier et al., 2021). The practical lesson is to show uncertainty and respect the data structure. A 95 per cent interval is not a promise that a candidate will perform inside that range on every future workload. It describes uncertainty under a model and sampling procedure.

If two intervals overlap, that alone is not a formal test of whether the difference matters. If they do not overlap, that also does not prove the observed gain is useful in production. Report a paired difference interval or an appropriate comparison, then interpret it against a predeclared decision margin. Statistical evidence and product significance are different questions.

Avoid false precision

Percentages with decimals can look more certain than the data allow. If a candidate passes 18 of 20 cases, saying “90.0%” does not make the estimate precise. Include raw counts, relevant denominators and intervals. Avoid ranking models by tiny differences when the estimated uncertainty and practical effect are both small.

Keep selection effects in mind. If you test 40 models and publish only the best score, the winner may be partly lucky. If you repeatedly tune on the same validation set, that set gradually becomes part of the development process. Use a fresh holdout for final decisions, and label exploratory analysis as exploratory. The companion article on holdouts and leakage explains this protection.

Track latency and operational variation too

Latency is a distribution. Record the test environment, concurrency, region, request size, streaming mode, warm-up policy and what event counts as completion. Report median and tail percentiles such as p95, plus timeout and error rates. A small number of very slow requests can matter even when the average looks good.

Cost varies with input length, output length, retries, caching, tool calls and service tier. Report both unit rates and the observed workload assumptions. When providers change pricing, a historical cost comparison remains meaningful only if it preserves the date and pricing basis. Do not infer cost from token count alone when calls or tools have separate charges.

A lightweight protocol

For a two-model comparison, a small team can use this sequence:

1. Freeze the dataset, prompt, tools, decoding and scoring versions.
2. Choose a paired set and any repeated-call subset before running.
3. Record raw output, status, latency, usage and errors for each call.
4. Calculate metrics per case and per run, keeping grouping intact.
5. Estimate uncertainty using a method suited to the sampling unit.
6. Report raw counts, point estimates, intervals and practically meaningful differences.
7. Inspect cases where candidates disagree and all critical failures.
8. Rerun a preselected sample if there is evidence of transient service error; document exclusions.

Do not discard inconvenient failures after the fact. Define exclusion rules in advance, such as invalid test fixture or provider outage. If a call fails because the system under test times out, that is part of the system result; if the evaluation harness itself mis-sent the request, correct the harness and rerun transparently.

Example interpretation

Suppose candidate A succeeds on 82 of 100 paired cases and candidate B on 85. In an illustrative analysis, B may win more paired cases, but the difference could still be too uncertain to support a release decision. Inspect whether the three-point gain comes from common routine cases or a high-risk slice, whether B introduces new critical errors, and whether repeated calls show a stable pattern. This example is hypothetical and conveys no measured model result.

A computable paired illustration

Marginal totals alone do not determine paired uncertainty. For this invented example, specify 80 cases both pass, 2 only A passes, 5 only B passes and 13 neither passes. A therefore passes 82 and B 85. Each case contributes a difference of +1 for a B-only pass, -1 for an A-only pass, or 0 otherwise. The mean difference is 0.03, or three percentage points.

The following standard-library Python computes a percentile bootstrap interval. It is an illustration, not a result from our editorial experiment. It assumes these 100 cases are independent and equally weighted representatives of the intended workload. It conditions on the recorded outcomes and does not measure extra variability from future stochastic model calls.

import random

rng = random.Random(20261022)
differences = [0] * 80 + [-1] * 2 + [1] * 5 + [0] * 13
replicates = sorted(
    sum(rng.choices(differences, k=len(differences))) / len(differences)
    for _ in range(20000)
)

def quantile(values, p):
    position = (len(values) - 1) * p
    lower = int(position)
    upper = min(lower + 1, len(values) - 1)
    return values[lower] + (position - lower) * (values[upper] - values[lower])

print("Observed B minus A:", sum(differences) / len(differences))
print("Illustrative 95% interval:",
      quantile(replicates, 0.025), quantile(replicates, 0.975))

The resampling preserves the A/B pair because it samples one within-case difference at a time. For correlated cases, resample the independent group instead, keep all cases in the sampled group and declare whether groups or cases receive equal weight. With very few independent groups or rare events, a percentile interval can be unstable; seek statistical review before using it as a consequential release gate. These assumptions matter more than printing extra decimal places.

The decision may be “collect more data” rather than declare a winner. Predefine a practical margin: perhaps the quality difference must exceed a threshold while cost or latency improves. The margin should reflect risk and the cost of delay. A team can also choose a safer candidate when evidence is ambiguous if the downside of a missed failure is asymmetric, while documenting that as a risk-based choice rather than a statistically proven superiority.

Preserve a reproducible record

Store the exact evaluation code, dataset version, prompt and tool definitions, model identifiers, request parameters, date, environment, logs and analysis output. Record deviations, missing responses and retries. Keep the raw item-level results so later analysts can test a different metric without reconstructing the experiment.

The central discipline is to separate what you observed from what you infer. “B scored three points higher on this run” is an observation. “B is better for our workload” is a broader claim that requires representative cases, stable scoring, controlled comparison and enough precision. When the evidence is weak, say so. When the evidence is strong for one slice but not another, publish the slice. This is how a benchmark supports a decision without pretending uncertainty has disappeared.