How to benchmark AI models for your own workload

A benchmark can tell you that one model scored higher on a published test. It cannot tell you, by itself, whether that model will solve your team’s problem within its budget, latency target and safety envelope. Choosing a model is an evidence problem: decide what “good” means for the workload, measure it with cases that represent that workload, and keep the comparison honest enough that someone else can reproduce it.

This guide gives you a practical path from public leaderboard to a decision you can defend. It is for technical founders, engineering leads and practitioners comparing language models or model-backed products. It does not rank current providers or prescribe one evaluation platform. The examples are hypothetical unless stated otherwise.

Start with the decision, not the leaderboard

Write down the decision the evaluation must support. “Which model is best?” is not a useful question until you name the job and the consequences of failure. For an invoice assistant, a decision might be: “Can we replace the current model with candidate B for extracting supplier, date, currency and total from English-language invoices, without increasing critical field errors, breaching a two-second p95 latency target or exceeding our per-document budget?”

This framing changes the experiment. It makes the unit of evaluation clear (a document and its extracted fields), defines what counts as a critical error, and prevents a strong result on unrelated chat tasks from settling the question. A useful evaluation objective includes:

• the user or workflow and the input distribution;
• acceptable and unacceptable outcomes, including severity;
• the system boundary being tested;
• operational constraints such as latency, cost, privacy and availability;
• a decision rule, including what happens when evidence is inconclusive.

Separate hard gates from preferences. A model that is cheaper but invents totals may fail a non-negotiable accuracy gate. Once every candidate clears the safety and quality floor, you can compare cost, speed and maintainability.

Use public benchmarks as a map, not a verdict

Public benchmarks are useful for narrowing the field, understanding capability families and spotting known limitations. Their scores depend on the tasks, data, prompts, scoring rules and model access used by the maintainers. MMLU, for example, was introduced as a 57-subject multiple-choice test spanning academic and professional topics; it was designed to probe broad knowledge and problem solving, not invoice extraction or your particular product workflow (Hendrycks et al., 2020). HELM makes the broader point that model evaluation covers many scenarios and desiderata: its original study selected scenarios and measured multiple dimensions, including accuracy, calibration, robustness, fairness, bias, toxicity and efficiency (Liang et al., 2022).

Treat a public score as evidence about the benchmark setup, as the cluster’s guide to reading public benchmarks explains. Ask what task it samples, whether the test is close to your use case, what version and prompt were used, which models were actually evaluated, and whether the score is reproducible. A leaderboard with a single aggregate can hide trade-offs and gaps in coverage. A benchmark may also age: systems change, task conventions shift and test items may become familiar through public release. These are reasons to use public results for screening and hypothesis generation, then test finalists on your own cases.

Your own evaluation should be deliberately small at first. Use the step-by-step guide to designing a task evaluation to turn the decision into a test set. Begin with a few dozen carefully chosen examples that reveal whether the task and rubric make sense. Then expand the set to cover common inputs, important subgroups and rare but consequential failures. There is no universal sample size that makes a benchmark representative; size depends on variation, error cost and the precision needed for the decision. If you have a long tail of unusual documents, an average score from a small random sample can conceal it.

Build a workload-specific test set

A good test set is not simply a pile of prompts. It is a documented sample of situations the product is meant to handle, paired with expected behaviour or a scoring procedure. Start with a task statement and an inclusion rule. For the invoice example, include normal supplier invoices, scanned documents, multiple currencies, credit notes, missing totals and line-item disagreements. Do not let “representative” mean only easy examples that happen to be available.

Create a case table with a stable ID, input or protected input reference, expected answer or acceptance criteria, relevant tags and provenance. Tagging matters: it lets you ask whether errors cluster around low-resolution scans, currencies or a specific language rather than relying on a global average. Keep sensitive customer data out of experiments unless you have an approved basis and controls; de-identification can alter the very cues being tested, so record any transformation.

For objective tasks, define exact or structured checks. For subjective outputs, use a rubric with observable criteria, examples and a way to handle acceptable variation. A single reference answer is often too narrow for open-ended writing. Prefer checking the properties that matter: required facts, groundedness in supplied context, prohibited claims and readability for the intended user. The companion guide to metrics and rubrics explains how to choose a scoring method that matches the task.

Keep development examples separate from a final holdout. Use development cases to refine prompts, tools and graders; use the holdout to estimate how the final configuration behaves on unseen examples. If you repeatedly inspect holdout failures and tune against them, the holdout has become another development set. Record access and changes, and refresh a portion of the set from new cases over time. The article on datasets, holdouts and contamination covers the main leakage paths and practical controls.

Decide what is inside the comparison

“Model A versus model B” can describe very different experiments. If one candidate receives a longer system prompt, a different tool schema or a larger output budget, the result compares configurations, not model weights alone. That may be exactly what you want: users experience the complete configuration. State the question clearly and choose one of two designs:

1. Controlled model comparison: hold prompt, tool definitions, retrieval, decoding settings, output limits and surrounding code constant wherever each provider supports a comparable setting. Record unavoidable differences.
2. Best practical configuration comparison: allow each model to use a reasonable, documented configuration, then compare the resulting systems under the same task and operational constraints.

The first helps isolate model effects. The second helps choose a deployable solution. Neither design is inherently superior; mixing them without labelling the difference makes results hard to interpret. For both, use the same test cases, preserve raw outputs, note model identifiers and API dates, and rerun after any material configuration change. Read how to make model comparisons fair before treating a leaderboard as a controlled experiment.

Measure more than answer quality

A useful scorecard usually has several columns, not one magic number. For the invoice workflow, track field-level exact match, document-level success, critical-field error rate, abstention behaviour, latency and cost per completed document. Add operational checks such as schema validity and whether failure handling preserves a human review path. The right measurements depend on the decision: a missing decimal point may be more serious than a harmless formatting variation.

For each metric, specify the numerator, denominator, unit, treatment of missing output and threshold. “Accuracy” is ambiguous when one output has four fields and another has one. “Cost per document” is ambiguous when retries, input tokens, tools or human correction are excluded. Keep component metrics visible even if you calculate a composite. If you must combine dimensions, publish the weights and the reason for them, and show whether a reasonable change in weights changes the ordering.

Human review can assess quality that scripts cannot reliably capture, but it needs a rubric and calibration. Give reviewers anonymised outputs, criteria with examples and a way to mark uncertainty. For pairwise review, randomise left/right order and include ties. Use two reviewers on an initial sample, inspect disagreements, revise unclear criteria, and reserve adjudication for consequential conflicts. Model judges can increase throughput, but check their agreement with human-labelled examples and test for position and style bias; they are instruments to validate, not ground truth. See human evaluation and model judges.

Make repeated runs and uncertainty visible

Language model outputs can vary across repeated calls, even when a request appears unchanged. Small samples also create uncertainty: an observed difference may reflect which cases happened to be in the set. Save run-level records and state whether you sampled cases, repeated calls, or both. A single run over 20 examples is not the same evidence as five runs over the same 20 examples.

For paired comparisons, score both candidates on the same items and inspect per-item differences. This controls for the fact that some cases are harder than others. Report an interval or another uncertainty summary alongside the average; do not read a tiny score difference as a real advantage when the uncertainty is larger. A statistical interval does not fix biased sampling, contaminated data or a flawed metric. It quantifies only the uncertainty represented by the calculation. The practical guide to repeatability and uncertainty offers a lightweight design.

If repeated sampling is affordable, repeat a subset of calls and distinguish retries from independent runs. Capture the dated model identifier where available; do not claim exact reproducibility when the service does not expose enough detail.

Put cost and latency into the same decision

A higher per-token price can still yield a cheaper task if the model needs fewer tokens, retries or corrections; a low unit rate can also hide a costly workflow. Calculate cost per successful task, including tokens, tools, retries and relevant review time. Pricing changes, so capture the date, region and service tier. The official pricing page shows that costs can include multiple token categories and tool calls.

Measure latency under the conditions users experience: warm or cold starts, concurrency, region, streaming or complete response, and the point at which a response is considered usable. Report a distribution such as median and p95, not only the mean. Keep reliability and timeout rates visible; a fast answer that fails on one request in ten may not be the right answer.

Evaluate an agent as a system

A benchmark of model completions does not automatically measure an agent. An agent can select tools, issue multiple calls, observe results, recover from errors and stop. Those actions depend on the model and the harness: tool descriptions, permissions, timeouts, memory, retrieval, environment and stopping rules. An agent comparison should test complete trajectories and task outcomes, not just the final text.

For a repository repair task, define the starting commit, issue, available tools, time budget, network policy, test environment and pass criteria. Score whether the required tests pass and whether the patch introduces regressions or unsafe changes. Record tool calls, retries, elapsed time and token use. Consider attempts that make partial progress, tasks that are impossible or underspecified, and tasks where doing nothing is safer than guessing. Benchmarks such as SWE-bench were created to evaluate systems on real software issues, but their results still depend on task curation and execution setup (Jimenez et al., 2023). A strong model can perform poorly in a weak harness; a capable harness can also conceal the contribution of the model. The dedicated piece on agent and tool-use evaluation expands the boundary.

Turn the result into an operating decision

Before the run, write a decision rule. For example: “Adopt candidate B only if its critical-field error rate is no more than one percentage point worse than the baseline, p95 latency remains below two seconds, and cost per successful document is at least 15 per cent lower; otherwise keep the baseline or run a larger test.” This is an illustrative rule, not a universal threshold. The acceptable margin must reflect risk and business context. If a safety-critical metric is uncertain, use a conservative gate or collect more evidence rather than average it away.

Publish a short evaluation record with the objective, data provenance, split, prompt and tool versions, model identifiers, date, run settings, scoring code, exclusions, results by slice, uncertainty, raw-output location and decision. Keep the benchmark alive: add incidents and newly observed edge cases, but do not silently change the test set and compare the new score with the old one as if nothing changed. The guide to maintaining evaluations over time shows how to version and refresh a suite without losing historical meaning. For the operating economics, use the companion guide to latency and cost.

You do not need a grand leaderboard to make a good decision. You need a test that represents the work, a scoring rule that rewards the right behaviour, a fair comparison and enough uncertainty reporting to know what the evidence can support. Public scores can tell you where to look. A well-maintained workload evaluation tells you what to do next.