Is that benchmark improvement real? Repeats and uncertainty

Candidate B scores higher than Candidate A. Before replacing A, ask what could have produced the gap: different sampled tasks, different outputs on repeated runs, a changed grader or an operating-condition difference.

Uncertainty analysis helps you describe the strength of the evidence. It does not turn an unrepresentative dataset into a representative one. The procedures here are recommendations for practical comparisons; all numerical examples are hypothetical and calculated, not observed benchmark results.

Separate the sources of uncertainty

Dataset sampling concerns which tasks you selected from the workload. Run variability concerns what happens when the same configuration attempts the same task again. Grader variability concerns how outputs receive labels. Operational variability includes changing load, tool behaviour and external data.

Design the experiment so you can tell these apart. Store task IDs, run IDs, candidate configurations, raw outputs and scorer versions. Freeze external evidence where possible, then run a separate test if live-environment behaviour is part of the question.

Ten repeats of one conversation can estimate consistency on that conversation. They do not provide ten independent conversations. If tasks share a template or customer, even different task IDs may be related. Carry grouping rules from the dataset design into analysis.

State which population the estimate concerns. A challenge set selected for difficult cases supports conclusions about those challenges. Its success rate is not automatically a forecast of ordinary traffic.

Keep the paired outcomes

Run candidates on the same cases and retain the result for each case. The pairing tells you where one succeeds and the other fails, which an aggregate score discards.

In an illustrative 100-case comparison, both candidates pass 75 cases and both fail ten. A alone passes five; B alone passes ten. A therefore passes 80 and B passes 85, giving B a five-percentage-point observed lead.

Only 15 cases distinguish their pass/fail results. Inspect those cases. Do B's extra successes occur in the target workflow, or do they come with regressions in a critical slice? The same overall gap can support very different decisions depending on that pattern.

Do not infer certainty from the lead. Nor should you compare two separate intervals and treat overlap as the only decision rule. Estimate the paired difference directly, using a method consistent with the sampling unit and design.

Put a single success rate in context

Suppose a configuration succeeds on 80 of 100 independent, similarly sampled binary tasks. A 95% Wilson interval is approximately 71.1% to 86.7%. That interval illustrates how much uncertainty a seemingly precise “80%” can conceal. NIST describes the Wilson method for proportions. NIST confidence interval guidance.

This is a calculated teaching example using a normal quantile of about 1.96. It assumes the binomial setting is a reasonable approximation. It does not account for biased task selection, shared conversations or a changing judge.

A frequentist 95% interval describes the long-run coverage of the interval procedure under its assumptions. It is not a promise that 95% of future answers will pass, nor a probability statement assigning 95% belief to this particular fixed parameter interval. Coverage interpretation in the SciPy documentation.

If every tested task passes, do not report proven perfection. Finite evidence leaves uncertainty about unseen failures. For critical requirements, combine an appropriate interval or bound with targeted failure testing and a decision about acceptable residual risk.

Estimate the difference with paired resampling

For a reasonably sized collection of independent tasks, one practical approach is a paired bootstrap. Sample task indices with replacement, retain both candidates' outcomes for each sampled index, calculate the success-rate difference and repeat to form a distribution.

An illustrative analysis recipe uses 10,000 resamples, a recorded random seed and a declared interval method. A simple percentile interval takes the 2.5th and 97.5th percentiles for a nominal 95% interval. This is a transparent starting method, not a universal best choice. SciPy documents bootstrap methods and the paired option, which resamples corresponding observations together. Bootstrap reference.

If cases are grouped by conversation, resample independent conversation groups rather than individual messages. Specify whether the target quantity gives equal weight to conversations or to tasks; calculate that same quantity within every resample. These choices can materially change the interpretation.

For small samples, rare failures or degenerate outcomes, a bootstrap can behave poorly. Obtain statistical help or use a method suited to that design before making a consequential claim. More resamples cannot manufacture diversity absent from the original cases.

Repeat the protocol without inflating the sample

Choose repeat counts and execution order before looking at a convenient lead. Repeat under the same timeout, retry, tool and scoring rules. Record every attempt, including transport failures according to the declared analysis policy.

Summarise variation within tasks and across whole runs. Report the mean success rate over repeats if that is the intended operational quantity, and separately show how often a task changes outcome. A configuration can have a reasonable average while repeatedly failing the same important class of request.

Do not pool all repeated outputs and claim they are independent new tasks. A hierarchical analysis may be appropriate when both task sampling and within-task variation matter. At minimum, disclose the nesting and avoid confidence claims based on an inflated case count.

Keep operational errors visible. If load causes timeouts, excluding them may answer a model-only question while hiding system reliability. Decide which question the report answers and provide separate breakdowns where necessary. The fair comparison guide covers configuration controls.

Decide what improvement would matter

Define a minimum practically useful change before running the final comparison. A tiny quality gain may not justify migration effort, higher latency or a new critical failure. Conversely, a moderate cost reduction may matter if quality remains within a predeclared acceptable margin.

A claim of non-inferiority needs an explicit margin and suitable analysis; absence of a statistically significant difference does not establish equivalence. If the interval spans meaningful improvement and meaningful harm, the comparison is inconclusive for that decision.

Account for the search process. If you try many prompts and candidates, then report only the best held-out score, you risk presenting selection luck as a stable result. Select on development evidence and evaluate the chosen configuration on a protected final set.

Set a stopping rule tied to precision, coverage or budget before testing. Do not repeatedly peek and stop the first time the preferred candidate leads. If sequential decisions are required, use a design that supports them and record it.

Finish with an honest decision statement: observed difference, uncertainty method, independent sample units, consequential regressions and action. Preserve that statement with the evaluation history. The pillar guide connects this evidence to quality, operating constraints and a reversible model choice.