Comparing Models Fairly: Prompts, Tools and Configuration

A bake-off is an experiment. The only thing that should differ between arms is the factor you are testing. Model behaviour is nondeterministic at every architecture level, from a single call to a multi-agent handoff (OpenAI, evaluation best practices). In practice, model comparisons are full of accidental advantages: one model gets a hand-written prompt, another gets a generic one; one gets a longer timeout, another is cut off at 30 seconds; one receives a cleaned retrieval context, another receives the raw HTML. The score then measures the configuration, not the capability.

This article is about making the comparison defensible, as the pillar guide argues. It defines a configuration manifest, distinguishes two legitimate prompt experiments, and audits the places where differences creep in.

The comparison is an experiment

Ask what claim you want to make. "Model A is better than Model B at our extraction task" is a claim about a task, a prompt strategy, a tool set and a cost envelope. If any of those differ, the claim is ambiguous.

List the confounders before you run:

• model version and provider;
• system prompt and formatting;
• temperature, top-p and maximum output;
• reasoning effort or mode;
• tool availability, descriptions and schemas;
• retrieval source, chunking, order and truncation;
• retry policy and timeout;
• concurrency and region;
• scoring rubric, judge and order;
• date.

If you cannot control a confounder, record it and disclose it. Disclosure is not as good as control, but it is far better than silence.

The configuration manifest

A manifest is a small file that travels with every result. A workable version:

run_id: 2026-10-24-extraction-v3
date: 2026-10-24
task_set: invoices-holdout-v2
models:
  - id: provider-a/flagship-2026-09
    snapshot: "2026-09-14"
    region: ap-southeast-2
    api: responses
  - id: provider-b/flagship-2026-10
    snapshot: "2026-10-02"
    region: ap-southeast-2
    api: chat
prompt:
  version: extraction-v3
  system: prompts/extraction-v3.txt
  temperature: 0
  max_output_tokens: 2000
  reasoning_effort: medium
tools:
  enabled: false
retrieval:
  enabled: false
runtime:
  timeout_seconds: 60
  retries: 2
  concurrency: 4
scoring:
  rubric: invoice-fields-v2
  judge: null
  order_randomised: true

The manifest is not bureaucracy. It is what lets someone else reproduce the result, and what lets you notice that a "model upgrade" coincided with a prompt change. A harness such as Inspect makes datasets, solvers, scorers and sandboxes explicit, which is exactly the discipline a manifest records (UK AI Security Institute, Inspect).

Two legitimate prompt experiments

There are two fair ways to compare prompts, and they answer different questions.

Identical prompt. Every model receives exactly the same system prompt, examples, formatting instructions and output contract. This answers: "Which model follows this prompt best?" It is the right design when the prompt is part of the product and will be held constant.

Provider-optimised prompt. Each model receives a prompt written in the style its provider recommends, with the same information and the same output contract. This answers: "Which model performs best when each is used as intended?" It is the right design when you are choosing a platform and will adopt its preferred patterns.

Both are valid. The mistake is to mix them: give one model a carefully tuned prompt and another a naive one, then call the result a model comparison. If you use provider-optimised prompts, give each model the same tuning budget on a development set, and record every version. If one model gets ten iterations and another gets one, you have measured the tuning effort.

Tool fairness

Tools are part of the agent system, and tool descriptions are prompts. A clearer tool schema can change the same model’s observed performance; measure that effect on the task. That is a real effect, but it is not a model difference.

Hold constant:

• the set of tools;
• tool names and descriptions;
• argument schemas and required fields;
• the sandbox and its permissions;
• error messages returned by tools;
• the maximum number of tool calls;
• whether the model uses native function calling, JSON mode or a parser.

Record the calling mode. A model that supports native function calling and a model that needs a text parser are not in the same condition, even with identical tool descriptions. If native calling is part of the platform's value, say so and treat it as part of the comparison. SWE-bench's Bash Only view, which runs every model in the same mini-SWE-agent environment, is a public example of holding the scaffold constant (SWE-bench leaderboards).

Context fairness

Retrieval is a common source of accidental advantage. One model may receive a shorter, cleaner context because it has a smaller window; another may receive the full document. That is a difference in the information available, not in reasoning ability.

Hold constant the retrieved documents, their order and the truncation policy. If a model cannot fit the context, record that as a constraint and evaluate it on the truncated input it will actually see. Do not quietly give it a better retriever.

Reasoning effort and cost

Reasoning models expose an effort or mode setting that trades tokens and latency for quality (OpenAI, reasoning models). Effort settings can change quality, token usage and latency; the direction and size depend on the model and task and should be measured. A comparison at different effort levels is not a model comparison; it is a comparison of two operating points.

Report both:

• matched setting labels: the same nominal label across models, with provider-specific semantics disclosed;
• matched cost: the setting that brings each model to a similar cost per task.

The first compares the recorded labelled configurations; it does not establish equal internal effort or computation. The second answers "which model gives the best quality for the money?" Both are useful. Neither is universal.

Retries and failures

Count retries, timeouts, refusals and parse failures as outcomes. A model that succeeds only on the third attempt has a different reliability and cost profile from one that succeeds first time. If you silently retry until success and report only the success rate, you have hidden the most important operational difference.

A practical rule: report first-attempt success and final success after retries separately. Report the retry distribution. Report cost including retries. The same applies to timeouts; a model that is fast because it gives up is not fast in a useful sense.

Scoring fairness

Use the same rubric, the same judge and the same order randomisation for every arm. Blind the judge to model identity. If the judge is a model, validate it against human labels, as described in human evaluation and model judges.

If you use a different judge for each model, you have introduced a new variable. If you use a judge that prefers a particular style, and one model has that style, the comparison is biased. The scoring pipeline is part of the experiment.

A confounder audit

Before trusting a bake-off, ask:

1. Are the model snapshots pinned and recorded?
2. Is the system prompt identical or intentionally adapted, and is that stated?
3. Did every model get the same tuning budget?
4. Are temperature, top-p and maximum output the same, or is the difference reported?
5. Are reasoning-setting labels and their limits disclosed, and is any matched-cost constraint stated?
6. Are the tools, descriptions and schemas identical?
7. Is the retrieved context identical in content and order?
8. Are retries and timeouts counted as outcomes?
9. Is the same scorer used, with order randomisation?
10. Is the date and region recorded?
11. Are failures included, or were they dropped?
12. Can someone else reproduce the run from the manifest?

A "no" is not automatically disqualifying. An unexplained "no" is. Where the audit reveals a difference, disclose it and consider a controlled rerun. An interval, described in repeatability and uncertainty, measures specified uncertainty; it cannot remove confounding.

Hypothetical: the six-point win that disappeared

Suppose Model B beats Model A by 6 percentage points on a 200-task extraction set. The team is ready to switch. The audit finds three differences: Model B received a 4,000-token output budget instead of 2,000, its tool schema had been rewritten by an engineer, and its timeout was 90 seconds instead of 30.

At matched budget, matched schema and matched timeout, the gap falls to 1 percentage point, inside the interval. The original result was a real measurement, but it measured a better-configured system, not a better model. The team could still choose Model B — the better configuration is a legitimate reason — but it should describe the decision accurately.

The point

Fairness is not a feeling. It is a set of recorded choices. The manifest, the two prompt experiments and the audit checklist turn a bake-off into evidence. The next article, on latency, token usage and cost, adds the economic dimension that makes the comparison useful to the business.

References

• OpenAI (2026). Evaluation best practices. Read source
• OpenAI (2026). Reasoning models. Read source
• SWE-bench (2026). SWE-bench Leaderboards. Read source
• UK AI Security Institute (2024). Inspect: Framework for Large Language Model Evaluations. Read source