Repeatability, Variance and Uncertainty in Model Evaluations

Imagine two models score 78% and 74%. One is "better". A week later the same two models, on a fresh sample, score 74% and 77%. Nothing about the models changed. The evaluation moved because the sample changed, the outputs varied, or the harness differed.

A score is an estimate, not a property. Reporting it without an interval implies a precision you do not have. This article gives the minimum toolkit for avoiding that mistake: standard errors, paired comparisons, clustered samples, power analysis and reliability metrics. It supplies the analysis step in the pillar guide. It is not a statistics course but a set of calculations that change a decision.

Why two numbers are not a comparison

Variation enters at several levels:

• Task sampling. The questions in the evaluation are drawn from a larger population. A different sample would produce a different score; see designing evaluation tasks and datasets.
• Output sampling. Generative models are nondeterministic. The same prompt can produce different answers, and one answer per task is one draw.
• Harness variation. Prompt formatting, tool descriptions, retries and parsing can change results.
• Provider variation. Models are updated, routed to different hardware, and sometimes silently changed behind an alias.
• Judge variation. If a model or a human scores the output, the scorer adds its own noise (human evaluation and model judges).

You cannot eliminate all of this, but you can measure it and avoid overclaiming. Benchmark variance has been measured across seeds and training runs, and simple format changes can move it (Madaan et al., 2024).

The standard error in one page

For n independent tasks with scores s_i, the mean is the estimate. Its standard error is the standard deviation of the scores divided by the square root of n:

SE = sd / sqrt(n)

For a pass/fail task, the standard error of the proportion is:

SE = sqrt(p * (1 - p) / n)

With adequate sample size and a suitable sampling model, a normal-approximation 95% interval is roughly the estimate plus or minus 1.96 standard errors. Near zero/one or with few binary cases, use an appropriate binomial method such as Wilson rather than an unqualified Wald interval. Report the number of tasks, the mean and the standard error together. "78% (SE 3.8%, n=120)" tells the reader more than "78%".

Miller recommends exactly this: compute standard errors from the data, report them beneath the mean, and include the number of questions (Miller, 2024). He also notes that bootstrapping is usually unnecessary when the central limit theorem applies, though it can help with complex estimators.

Paired comparisons

When both models see the same tasks, compare them task by task. For each task, compute the difference in score between Model A and Model B, then take the mean and standard error of those differences.

The gain is variance reduction. If the models succeed on the same easy tasks and fail on the same hard ones, their scores are positively correlated, and the difference is less noisy than either score alone. Miller shows the paired standard error can be materially smaller and recommends paired analysis wherever the same questions are used.

A worked hypothetical: 120 tasks, Model A at 78%, Model B at 74%. If the per-task scores have a correlation of about 0.5, the paired standard error of the difference is roughly 3.9 percentage points. The 95% interval runs from about −3.6 to +11.6 points. It includes zero. The data do not establish a winner.

The same arithmetic shows the sample-size problem: detecting a 4-percentage-point difference with 80% power at that variance and correlation needs roughly 900 tasks. These numbers are illustrative. The method is the point: decide what difference matters, estimate the variance, and size the evaluation in advance.

Clustered tasks

Many evaluations are not independent draws. Reading-comprehension sets ask several questions about the same passage. Multilingual sets ask the same question in several languages. Agent evaluations run several turns in one conversation, or several files in one repository. Those tasks are correlated within clusters.

Ignoring clustering understates the standard error. Miller reports clustered standard errors up to about three times larger than naive ones on real data. Compute the standard error at the cluster level, or use an estimator that accounts for within-cluster correlation. If unsure, sample whole clusters and report both counts.

For equal-sized clusters, the design effect is approximately 1 + (m − 1) × rho, with m items per cluster and within-cluster correlation rho. For 500 questions in 50 ten-question documents, the effective size ranges with that dependence; at rho=1 it is about 50, while weaker correlation retains more information. Plan with observed or justified dependence rather than equating every clustered set to its number of groups.

Repeated trials

Run each task more than once. If each task is sampled k times and the score is the mean of those k answers, the conditional variance falls roughly as 1/k. The total variance is the sum of task-level variance and output-level variance, so repeated trials help until task-level variance dominates.

Miller advises choosing k so that output variance divided by k is small relative to task variance. In a simple binary example, two samples reduce total variance by about a third, four by about a half, and the upper limit is about two-thirds. More trials help, with diminishing returns.

Do not lower the sampling temperature just to reduce variance. Temperature changes the tested configuration and may change both the mean result and its variance. It does not establish reliability under another configuration. If you want to study the model at a lower temperature, do so deliberately and report it.

pass@k versus pass^k

These two metrics look similar but answer different questions.

pass@k is the probability that at least one of k attempts succeeds. It measures capability: can the system ever solve this task? The HumanEval estimator is 1 - C(n-c, k) / C(n, k) for n samples and c correct, with n ≥ k; calculate per task and average across tasks (Chen et al., 2021).

pass^k is the probability that all k attempts succeed. It measures reliability: will the system work every time? τ-bench introduced it because agents can succeed intermittently, and reported pass^8 below 25% in its retail domain even when single-attempt success was under 50% (Yao et al., 2024).

For customer-facing agents, report single-attempt success alongside repeated-success reliability when consistency matters. A system with 70% single-attempt success but only 30% success across all five attempts has substantially weaker five-attempt reliability; the two figures answer different questions. See evaluating agents and tool use.

Power analysis

Power is the probability that an experiment detects an effect of a given size if that effect is real. An underpowered evaluation will often miss a real difference, and when it does detect one, the estimated size may be exaggerated. Card and colleagues found that underpowered experiments were common in NLP, and that typical machine-translation test sets of 2,000 sentences had about 75% power to detect a 1-BLEU difference (Card et al., 2020).

A simple planning formula for a two-model comparison is:

n ≈ (z_alpha/2 + z_beta)^2 * variance_of_difference / effect_size^2

For 5% significance and 80% power, the first term is about 7.85. Miller's worked example, with stated assumptions, implies about 969 independent questions to detect a 3-percentage-point difference. The exact number depends on variance and correlation; that figure is conditional on the example’s variance, correlation and desired power, not a universal decision-grade starting size.

If the evaluation is fixed at 120 tasks, invert the formula to compute the minimum detectable effect. That tells you what the evaluation can and cannot decide. It is better to say "this test can detect a 10-percentage-point difference" than to pretend a 2-percentage-point difference is meaningful.

A reporting table

A compact table is enough:

The success percentages and interval illustrate one attempt on each of 120 independent tasks. The table uses one trial per task; repeated-trial planning above is a separate design choice. No interval was calculated for the illustrative cost or latency rows. The interval is the important column; the cost and latency rows show why an inconclusive quality difference may still produce a clear decision.

Decision rules

• Pre-register the threshold and the comparison.
• Treat an interval that crosses the threshold as "not yet decided".
• Prefer effect sizes and cost-quality curves to rank order.
• Do not compare the best of many runs; report all runs or a pre-specified aggregate.
• Adjust for multiple comparisons when you test many models or many metrics, or replicate the finding on a fresh set.
• Separate statistical significance from practical significance. A real 1-percentage-point gain may be irrelevant.

Failure cases

Twenty-task comparisons. The interval is so wide that almost any difference is compatible with the data.

Single-run claims. One sample per task measures one draw, not the model.

Seed shopping. Running until a favourable seed appears, then reporting only that run.

Temperature tweaks. Lowering temperature to make the numbers look stable, without reporting the change.

Ignoring clusters. Treating 500 questions about 50 documents as 500 independent observations.

Best-run reporting. Publishing the maximum rather than the distribution.

The point

Uncertainty is not a caveat to add at the end. It is part of the result. A score with an interval tells you what you know; a score without one invites you to believe more than the evidence supports. The next article applies this discipline to the comparison itself: comparing models fairly.

References

• Card, D., Henderson, P., Khandelwal, U., Jia, R., Mahowald, K. and Jurafsky, D. (2020). With Little Power Comes Great Responsibility. EMNLP 2020. Read source
• Chen, M., Tworek, J., Jun, H. and others (2021). Evaluating Large Language Models Trained on Code. Read source
• Madaan, L., Singh, A. K., Schaeffer, R., Poulton, A., Koyejo, S., Stenetorp, P., Narang, S. and Hupkes, D. (2024). Quantifying Variance in Evaluation Benchmarks. Read source
• Miller, E. (2024). Adding Error Bars to Evals: A Statistical Approach to Language Model Evaluations. Read source
• Yao, S., Shinn, N., Razavi, P. and Narasimhan, K. (2024). τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains. Read source