Compare AI models fairly across prompts, tools and settings

“Use the same prompt” sounds like a complete fairness rule. It is only the beginning. Candidates may use different chat templates, support different controls, receive different context or spend different amounts of computation before answering.

A fair comparison makes those choices explicit and matches them to the decision. The procedures here are editorial recommendations. They describe how to design an experiment, not measured evidence that any candidate is better.

State which comparison you mean

A fixed-harness comparison changes the model while keeping the surrounding task and application behaviour as stable as practical. It helps answer whether a model substitution improves the existing workflow. Differences that cannot be controlled must be recorded.

A deployment-optimised comparison asks which complete configuration performs best within operating constraints. Each candidate may receive a suitable prompt, effort setting or workflow adaptation, subject to a declared development budget. It evaluates the deployable package.

Neither experiment should pretend to be the other. If one candidate gets an optimised prompt and another gets the incumbent's prompt, the result may be useful for a migration decision, but it does not isolate model capability.

Write the comparison type in the run plan and result title. For a hypothetical support assistant, you might first test direct substitution on supplied policy passages, then tune two configurations on development cases and compare the frozen finalists. Report both experiments separately.

Freeze the conditions that define the task

Hold the user requests, relevant evidence, permitted actions and scoring criteria constant. Use the same policy and order snapshots. A candidate given the missing delivery date is solving a different task from one required to ask for it.

Preserve message roles and request structure where interfaces permit it. If one system uses a special instruction channel and another needs a different formatting convention, document the adaptation. Equal literal text does not guarantee equal rendered model input.

Prompt formatting itself can affect results. Sclar and colleagues found substantial sensitivity to meaning-preserving format choices in their studied models and few-shot tasks, with weak transfer of preferred formats between models. That finding motivates a sensitivity check; it does not establish the size of the effect for every current model. Prompt-format study.

Test a small, predeclared set of plausible formats on development data if formatting is material. Do not search formats on the final holdout and then report the winning format as if it had been selected in advance.

Give tuning a declared budget

For a deployment-oriented comparison, decide how much development effort each candidate receives. Budget by a meaningful resource: number of prompt trials, human time, API spend or a combination. Keep a tuning log, including unsuccessful changes.

Equal numbers of trials do not guarantee equal engineering effort, but a disclosed budget is better than invisible optimisation. Record prior investment in the incumbent. A candidate benefiting from months of accumulated examples is not starting from the same position as a newly integrated alternative.

Use separate development and holdout data. Freeze the final prompt, tools and configuration before the decision run. If a holdout defect requires repair, document it and apply the correction consistently.

Make stopping rules explicit. Stop tuning when the planned budget ends or the declared requirement is met on development evidence, not when a preferred candidate briefly leads on the final test.

Match capabilities without inventing equivalence

Record temperature, sampling controls, output limits and reasoning settings where available. Mark unsupported settings as unsupported. A value called “temperature” on two interfaces is not evidence of identical output distributions.

Similarly, an identical output-token cap does not necessarily impose an equal total computation budget. Reasoning behaviour and billing categories may differ. For OpenAI reasoning models, the documentation states that reasoning tokens occupy context and are billed as output tokens even though they are not visible as normal answer text. Reasoning model documentation.

For the operating decision, compare quality under a cost or latency envelope as well as under declared settings. Report the observed usage instead of claiming a perfect internal-compute match you cannot establish.

If a candidate cannot fit the required context, record the limitation. Do not silently truncate its input while describing the workload as identical. You can test an adapted system with retrieval or summarisation, but that becomes a separate configuration.

Control tools and retrieval

Use the same tool semantics and data snapshots when isolating the model. Save tool names, descriptions, schemas, error responses and permission boundaries. Tool definitions are part of the instructions a candidate receives.

For retrieval, distinguish identical retrieved passages from each system's native retrieval pipeline. The first isolates answer generation given evidence. The second evaluates end-to-end retrieval and generation. Both are useful if labelled correctly.

Set request and task timeouts, retry limits and feedback rules. Does a retry see the previous failure? Can the system call a verifier and revise an answer? Additional attempts and feedback can improve a package while increasing cost and changing what the result means.

For agents, reset initial state between cases and inspect actions, not only final text. The agent evaluation guide explains why tools and state make this a system experiment.

Save a run manifest someone else can use

Record a run ID, timestamp, candidate identifier, provider or local endpoint, exact model version when exposed, request parameters and application commit or equivalent version. Include prompt, dataset, context-builder, tool and scorer versions.

Store the execution environment, client version, concurrency, region where relevant, cache policy and error-handling rules. For a locally hosted model, add the checkpoint identifier, tokenizer, quantisation, inference engine and hardware details that affect the run.

Retain actual requests and responses where permitted, with a documented redaction policy. Hashes help identify artefacts, but a hash without retained content cannot reconstruct a prompt. Do not put secrets in a reproducibility bundle.

A maintained harness can reduce ad hoc implementation differences. EleutherAI's Language Model Evaluation Harness provides a shared framework for evaluating language models. Using one still requires identifying the task and configuration you ran. Harness repository.

Report exceptions as results

Keep rate limits, transport errors, unsupported features and parser failures in the execution record. Explain whether each contributes to task success, model-only scoring or a separate operational measure. Never erase a bad run simply because it is inconvenient.

Interleave or randomise candidate execution where practical so a provider slowdown does not affect only one candidate. Record the ordering. Repeat under the same declared protocol to investigate run variability, using the uncertainty guide.

Show sensitivity results separately from the primary comparison. If modest prompt changes reverse a ranking, that is useful evidence about the stability of the decision. It should narrow your conclusion rather than disappear in a footnote.

Finally, pair quality with latency and cost, and read external claims with the leaderboard audit. The pillar guide uses these records to support a bounded choice: this configuration, under these conditions, meets this workload's needs.