Compare AI models fairly when prompts and tools differ

A comparison is fair when its rules match the question it claims to answer and are applied transparently. Using the same prompt everywhere can isolate a model change, but may disadvantage a model that expects a different format. Tailoring every configuration can reveal the best deployable system, but makes it harder to attribute the result to the model alone. Choose the question before you choose the rules.

Take a fictional assistant that answers policy questions using retrieved excerpts. A team wants to compare two candidates. Both need the same protected questions and a common definition of success, but the rest of the design depends on whether the team is studying model substitution or selecting an entire production configuration. The examples below are illustrative.

Choose the comparison you intend

For a controlled substitution, hold the prompt, retrieved context, tool interface, output contract and scoring as constant as practical. Change the model and record any unavoidable compatibility adaptations. This makes diagnosis easier: a difference is more plausibly associated with the substitution. It does not guarantee that either model has been used optimally.

For a best-system comparison, permit model-specific prompts or configurations, then compare the complete systems. Give each candidate a declared tuning budget, access to the same development cases, equivalent engineering time where feasible and the same final holdout. Record the resources used. The result supports a decision about those systems, not a pure statement about the models' intrinsic capabilities.

Both are useful. Run a controlled comparison first to understand the baseline, then a bounded best-system comparison if the deployment decision warrants it. Do not quietly switch between these interpretations after seeing which candidate wins. HELM illustrates why common scenarios and documented conditions matter for comparisons; its broad coverage still does not substitute for a local workload.

Freeze a run manifest

Record the exact model and endpoint identifiers available, date and time, prompt text and version, examples, input formatting, context, output schema and generation settings. Include tool descriptions, tool versions, retrieval index and corpus version, maximum steps, timeout, concurrency, retry policy and selection rule. Note unsupported settings or moving aliases. Store outputs and errors under the appropriate privacy controls.

If one model requires a different message format, record the transformation and show that it preserves the task and evidence as closely as practical. If one supports a tool interface unavailable to another, decide whether to compare under a common interface or evaluate each complete system with its native interface. Either can be legitimate; label the result accordingly.

Make the scoring rule independent of candidate identity. Use the same case-level rubric, reference evidence and grader version. Blind and randomise human review where possible. If a model judge is used, check its agreement with human labels for each candidate, since a judge can prefer one style. The judge guide explains calibration and order checks.

Control the resources that can change outcomes

An agent allowed ten attempts and feedback from tests has a different opportunity from one allowed a single response. Fix or disclose attempt count, token and time budgets, tool calls, external search, selection of final answer and human intervention. Report total resources used, including failed attempts. If one candidate cannot use a resource, state the limitation rather than pretending the setups are identical.

For the fictional assistant, one fair controlled test could supply the same frozen excerpts to both candidates and ask for an answer with citations. A second end-to-end test could let each system retrieve from the same frozen corpus using its selected retrieval configuration. The first investigates response generation given evidence; the second evaluates the retrieval-and-answer system. If results differ, inspect retrieved passages before attributing the failure to reasoning.

Run paired cases close in time, and randomise or interleave execution order when service load or drift may matter. Apply the same concurrency and timeout rules. Track rate limits, failures and retries. A candidate with more missing outputs should not look better merely because only completed responses were graded. Report attempted, completed, graded and excluded counts.

Treat prompt tuning as development

Use a development set to refine instructions. Record prompt revisions, rationale and the number of iterations for each candidate. Avoid tuning one candidate extensively while giving the other a cursory prompt if the claim is about their best deployable systems. Equal time is not a perfect measure of effort, but a declared budget and change log make asymmetry visible.

Freeze each final configuration before opening the holdout. If the team revises it after seeing holdout failures, report the original result and treat the revision as a new development cycle with a fresh decision set. The holdout guide describes why repeated use changes the role of a test set.

Include cases where the best response is a clarification or refusal, not just cases with a direct answer. Give each candidate the same authority and side-effect constraints. An apparently high success rate achieved by taking an unauthorised action is not a fair measure of a system intended to request approval first.

Publish a comparison record

A useful record states the decision, comparison type, case and rubric versions, each configuration, tuning and runtime budgets, run dates, quality by slice, failures, latency and cost. Add uncertainty and an honest description of any uncontrolled difference. If a provider changes an alias mid-run or a tool has an outage, preserve that fact and decide whether the run supports the intended inference.

Fairness is not a checklist that makes all results universal. It is a commitment to make the comparison answerable and reproducible enough for its stated purpose. Use repeatability and uncertainty to assess the strength of the difference, and the benchmarking overview to connect it to the wider decision.

Handle capability gaps explicitly

Suppose candidate A supports a native structured tool call and candidate B requires a text wrapper. A common text interface can answer how the models behave under that shared constraint. A native-interface comparison can answer which complete system works better with the capabilities actually available. In the latter, validate both outputs against the same semantic contract and report the engineering needed to support each interface. Neither result proves the other.

If a candidate cannot fit a case within its available context or cannot access a required tool, do not silently omit that case from its denominator. Record the limitation and decide in advance whether it is a failure for the intended deployment or a separate capability exclusion. The baseline must receive the same treatment under the chosen rule. Where the deployment can route long cases elsewhere, evaluate that routing and its cost as part of the system. Make the scope of any conclusion clear: “under the common wrapper” and “with the tailored tool stack” are different claims, even if the same model names appear in both tables.

A comparison manifest to fill before a run

This completed outline is a teaching example. For a real run, replace generic descriptions with exact versions and authorised evidence before opening the holdout.

Return to the Terra cluster guide · Inspect the four-way results and evidence.