How to compare AI models fairly

A comparison can be perfectly consistent and still be unfair to the question you wanted to answer. Give one candidate more context, a better tool description, a higher output limit or more retries, and the winner may reflect those choices rather than a model difference. Sometimes the best question is model-only; sometimes it is which deployable configuration works best. State which experiment you are running.

This article provides a practical control plan, linked to the model benchmarking pillar.

Choose the comparison question

A controlled model comparison tries to hold the surrounding system constant so that model identity is the main changed factor. Use it when you want to understand relative model behaviour under a shared interface. A best-configuration comparison lets each model use a documented prompt and settings that suit its interface, then compares the complete solutions. Use it when choosing what to deploy. Results answer different questions; label them distinctly.

Model APIs are not always symmetric. One may support a setting another does not, or expose different sampling controls, context sizes and tool semantics. Record the mismatch and avoid implying a pure model effect. If the product will use each provider’s recommended configuration, that asymmetry is part of the deployment decision. If the evaluation is meant to isolate model quality, constrain candidates to a common supported subset and disclose the compromises.

Freeze the surrounding system

For a controlled comparison, pin the task instructions, system and developer prompts, examples, retrieval corpus, tool schemas, tool results, parsing code, safety filters, output limits and retry policy. Capture versions or hashes for files that may change. Send the same input cases and use the same scoring code. Log all requests and responses.

Prompts can change the task. A prompt that tells one candidate to “show every reasoning step” may increase token use and expose different content than a concise instruction. Apply equivalent instructions where possible, but do not assume identical text creates identical conditions across different model families. Document what was held constant and what was adapted. Avoid requesting hidden chain-of-thought; score the observable output and task behaviour.

Tools can dominate the result. If a candidate can call search while another is offline, the test compares access to information as well as model behaviour. Give equal tool availability, or explicitly compare complete product configurations where the tool difference is intended. Freeze tool versions, permissions, result quality, timeout and error handling. A tool failure should be logged, not quietly removed from the denominator.

Control tuning and selection

Do not tune one candidate extensively on the test set while leaving another at defaults. Give candidates the same development budget, use a separate validation set for tuning, and reserve a final holdout for the decision. Record how many variants were tried. If provider-specific tuning is part of the deployment comparison, define an equivalent time or compute budget and retain the final settings before the holdout run.

When possible, randomise run order so time-of-day load or transient service conditions do not systematically favour one model. Interleave paired requests rather than running all of A and then all of B if provider drift is plausible. Repeat selected cases and preserve raw results. The uncertainty guide explains paired analysis and intervals.

Make the runtime comparable

Match context available to the model, not just the prompt string. Ensure retrieval returns the same passages, truncation rules are consistent and the same documents fit within each candidate’s allowed context. If one model receives the entire corpus and another only a summary, note the change. Fix temperature or sampling settings where the APIs support a valid common comparison; if not, state the defaults and why they were selected.

Set a common user-facing stopping rule. One candidate may stream quickly but complete slowly; another may provide a usable answer sooner. Decide whether the metric is time to first useful output, total completion or both. Enforce the same timeout and retry policy if the question is controlled comparison. In a best-configuration test, allow different policies only when they are part of the actual product plan and include their costs.

Compare formats and safety behaviour

A JSON task should be evaluated with the same schema and downstream parser. Report whether formatting failures are counted as model failures, retried or repaired. A provider-specific constrained-output feature may be a legitimate advantage in a deployment comparison, but it changes the system boundary. Separate raw model formatting from the reliability of the final pipeline.

Apply the same safety policy and prohibited-case set to every candidate. Do not compare one model’s refusal rate against another’s without distinguishing correct refusals from unnecessary refusals. A model that refuses a benign request may be less useful; one that complies with a dangerous request may be less safe. A single aggregate safety score can hide both. Have qualified reviewers adjudicate ambiguous high-impact cases.

Example: summarising incident reports

Imagine comparing two models for incident summaries. The controlled test uses the same redacted reports, instructions, output schema, source references and maximum completion budget. Both candidates are evaluated on the same cases. Quality is assessed for factual support, important-event coverage and actionability; latency and cost are measured under the same concurrency. This isolates behaviour under the specified conditions.

A second experiment may allow each model a different prompt and provider-native structured output, selected on a development set. The resulting systems are then compared on an untouched holdout. That answers which configured system looks more deployable under the team’s constraints. Both experiments are useful, but a result from the second cannot be described as the effect of model weights alone. This example is hypothetical.

Publish the configuration ledger

For every result, report model name and identifier as exposed, date, provider endpoint or region, prompt version, tool setup, decoding parameters, maximum output, retrieval snapshot, retry policy, evaluation set and scoring version. If exact backend version is unavailable, state that limitation. Do not rely on screenshots of a dashboard as the only record.

Capture pricing and availability at the research date, with service tier and token categories where relevant. These details change. A model’s cost per successful task can differ from its per-token price because output lengths, retries and tool calls differ. The guide to latency and cost lays out the calculation.

Interpret the result within its boundary

If the candidates perform similarly under a controlled test, that does not prove they are interchangeable in a production system with other prompts and tools. If one wins under a best-configuration test, that does not tell you which component caused the gain. Slice results by task conditions and inspect failures. Use a confidence interval or paired comparison rather than overinterpreting a tiny average difference.

A fair comparison is not a claim that every candidate faces identical internals. It is a transparent experiment where choices are aligned to the question and their effects are visible. Keep settings fixed when you want attribution; allow documented optimisation when you want a deployable solution; use separate result labels for the two. Then ensure the test data, scorecard and uncertainty are sound enough to support the decision.

Evidence for the evaluation design

HELM: Holistic Evaluation of Language Models documents scenario coverage, multiple evaluation dimensions and standardised adaptation. It supports making the tested task and configuration explicit; the sampling and release choices above remain design recommendations for your workload.