Evaluating AI Models and Agent Systems: A Practical Guide for Technical Teams

Consider an illustrative team that picks the model at the top of a leaderboard, wires it into a support workflow, and ships. Two weeks later the complaints arrive. The model writes beautiful prose but ignores the refund policy. It extracts the right order number on Tuesday and the wrong one on Thursday. It costs four times the estimate because every request carries a long system prompt and three tool calls. Nothing in the benchmark table predicted any of this.

The failure is not that the team chose a bad model. It is that they treated a benchmark score as a property of the model, when it is evidence about a system on a sample of tasks under a particular configuration. Change the tasks, the prompt, the tools or the date, and the evidence may no longer apply.

This guide sets out a repeatable way to make those decisions: what to measure, how to sample, how to judge, how to quantify uncertainty, and how to put cost and reliability on the same scorecard. It is written for technical leaders who need to choose or defend a system, not for evaluation researchers. Claims with citations are sourced; recommendations are editorial judgement; numbers marked hypothetical are there to demonstrate arithmetic, not to report a result.

What a benchmark score actually claims

A public benchmark is a fixed task sample, a harness, a metric and a date. Each component is part of the claim. MMLU, for example, is 57 tasks covering subjects from elementary mathematics to law, and it was designed to test broad academic knowledge and problem solving (Hendrycks et al., 2021). BIG-bench assembled 204 tasks from 450 authors across 132 institutions and compared models with human expert raters (Srivastava et al., 2022). HELM went further, measuring seven metrics across 16 core scenarios and 42 scenarios in total, under standardised conditions, because accuracy alone hides trade-offs in calibration, robustness, fairness, bias, toxicity and efficiency (Liang et al., 2022).

These are serious instruments, but a multiple-choice test does not tell you whether a model can hold a JSON schema for 40 turns, use your internal tools, recover from a failed API call or resist a prompt injection in a retrieved document. When frontier scores cluster near the ceiling, the test stops discriminating: a two-point gap may be noise, or an artefact of prompting. A benchmark can be well designed and still be saturated, contaminated or irrelevant to your distribution.

Leaderboards add another layer. Chatbot Arena measures pairwise human preference; it had more than 240,000 votes at the time of its paper and good agreement between crowd and expert raters (Chiang et al., 2024). But preference is not task fitness, and the ranking process itself can be distorted. An analysis of Arena data documented private testing, selective disclosure of results, and large asymmetries in how often models are sampled, concluding that these dynamics can reward overfitting to the arena rather than general quality (Singh et al., 2025). An assessment of 24 benchmarks against 46 lifecycle best practices found large quality differences and that most did not report statistical significance or allow easy replication (Reuel et al., 2024).

Use public benchmarks to shortlist, sanity-check a vendor claim and learn the vocabulary of a capability area. Do not use them to make a deployment decision.

The system is the unit of evaluation

A model is one component. The system also includes the prompt, tool definitions, retrieval, the scaffold that decides when to call a tool, policy, sandbox and runtime. Each adds a place for behaviour to vary.

OpenAI's evaluation guidance makes this concrete by separating single-turn interactions, multi-step workflows, single-agent systems and multi-agent systems. Each architecture introduces a new source of nondeterminism: instruction following and functional correctness at the single-turn level; tool selection and argument precision once tools exist; handoff accuracy once several agents exist (OpenAI, evaluation best practices). If you evaluate only the model endpoint, you have not evaluated the system your users will touch.

Public agent benchmarks illustrate the point. SWE-bench evaluates 2,294 real GitHub issues from 12 Python repositories, and its leaderboard now reports several subsets, including a 500-instance Verified set filtered by humans and a Bash Only view that runs every model in the same mini-SWE-agent environment (Jimenez et al., 2023; SWE-bench leaderboards). That standardised scaffold is the point: without it, a score confuses the model with the harness. GAIA found that humans answered 92% of its questions while GPT-4 with plugins answered 15%, a gap that is about tool use and multi-step execution as much as language ability (Mialon et al., 2023). τ-bench compares the final database state after a simulated user conversation and introduced pass^k to measure whether an agent succeeds on all k attempts; even strong function-calling agents succeeded on fewer than 50% of tasks, and pass^8 was below 25% in the retail domain (Yao et al., 2024).

Those results are not arguments that agents are bad. They are arguments that a model benchmark is the wrong instrument for a system that plans, calls tools and changes state.

A six-step evaluation loop

The loop below is deliberately small: enough for a defensible decision without building an evaluation platform.

1. Name the decision and the threshold. Write down the choice, the options and what counts as good enough. "Choose between two models for invoice extraction, where a 2% error rate is acceptable and a missed line item costs more than a false positive" is a decision. "Evaluate models" is not. The threshold determines the sample size, the metric and the level of uncertainty you can tolerate.

2. Build a task sample that resembles the workload (designing evaluation tasks and datasets). Draw from production logs, support tickets, documents or code review history, then stratify by intent, frequency, difficulty and risk. Include happy paths, edge cases and adversarial cases. Record the provenance and date of every task. A small convenience sample may expose failures, but does not establish representative performance.

3. Choose a scorer that matches the decision. Prefer an executable oracle: an exact answer, a passing test, a correct database state, a valid schema. When the output is open-ended, use a rubric with observable criteria, and calibrate it against human labels. A metric is a measurement instrument; it needs its own validation.

4. Design the run. Hold the configuration constant except for the factor you are testing. Record the model snapshot, prompt version, temperature, reasoning effort, tools, retries, timeout and harness version. Run multiple trials per task, because a single sample of a nondeterministic system is a single draw.

5. Analyse the result with uncertainty. Report the number of tasks and trials, the mean and its standard error, the paired difference between candidates, and the interval. Add cost, latency and failure categories. A difference that is smaller than the noise is not a result, however impressive the decimal places.

6. Decide and monitor. Choose, document the conditions, and set a review date. Put the evaluation into a regression suite. Watch production for the failures the test set missed, and convert incidents into new tasks.

Go back if the sample proves unrepresentative, or if the scorer disagrees with experts; fix the instrument before running more models.

The failure modes that decide outcomes

Most bad model decisions come from a small set of mistakes.

Public benchmark overfitting. A model that ranks highly on a general test may be mediocre on your task, and a team may tune its prompts against a public test until the score stops meaning anything. The fix is a private, workload-specific sample (public benchmarks versus workload evaluation).

Contamination and leakage. Test items may appear in training data, be paraphrased into it, or be reused so often that they become development data. String matching is not enough: rephrased samples can bypass n-gram filters, and a study found 8–18% HumanEval overlap in two large pretraining corpora (Yang et al., 2023). A sealed holdout and fresh, date-stamped tasks are the practical controls (contamination, leakage and holdouts).

A weak or unvalidated scorer. Exact match punishes legitimate paraphrase; ROUGE rewards overlap rather than correctness; an LLM judge may prefer longer answers or its own family's style. The scorer needs a validation set and an error audit (objective metrics and rubrics; human evaluation and model judges).

Underpowered comparisons. A 4-percentage-point gap on 120 tasks can be indistinguishable from noise. Miller's framework shows how to compute the standard error, use paired differences, account for clustered tasks and plan the sample size in advance (Miller, 2024). If the interval crosses your decision threshold, you have not decided yet (repeatability, variance and uncertainty).

Unfair configuration. One model gets a hand-tuned prompt, a longer timeout, more retries or a better tool schema. The comparison then measures the configuration, not the model. Record everything and run the same experiment for every candidate (comparing models fairly).

Cost blindness. Accuracy-only evaluation rewards systems that are needlessly complex and expensive. Kapoor and colleagues argue for optimising accuracy and cost jointly, and show that agent benchmarks often lack adequate holdouts and standardised reporting (Kapoor et al., 2024). A model that is 3 percentage points better at ten times the cost may be the wrong choice (latency, token usage and cost).

Agent unreliability. A single successful run is not reliability. Report pass^k, the number of steps, the failure mix and cost per successful task. Report both: 70% single-attempt success does not establish consistent success across repeated runs; only 30% of tasks succeeded on all five attempts in this example (evaluating agents and tool use).

Evaluation decay. Models are updated, prompts change, judges drift, prices move and holdouts get reused. An evaluation is a living asset with an owner and a review cadence, not a report (interpreting results and maintaining evaluations).

A minimum viable evaluation

A useful first evaluation does not need a research team. One possible exploratory starting plan, to adapt after precision and risk analysis, is:

• 100–300 tasks, stratified across three to five workload-relevant categories, with a sealed holdout of at least 30%;
• two to four candidate systems, including the current baseline;
• three trials per task, with the same tasks and the same scorer for every candidate;
• one objective metric or rubric gate, plus a small set of failure categories;
• 50–100 human-labelled items to calibrate any model judge;
• cost and latency measured on the same runs;
• a written decision rule and a review date.

Consider a hypothetical comparison. Model A scores 78% and Model B scores 74% on 120 tasks. If the models tend to succeed on the same easy items, the paired standard error of the difference might be around 3.9 percentage points, giving a 95% interval of roughly −3.6 to +11.6 points. The interval includes zero, so the data do not establish a winner. To detect a 4-percentage-point difference with 80% power at the same variance and correlation, the arithmetic implies roughly 900 tasks, not 120. These are illustrative numbers, not measurements; the point is that small evaluations are good for finding gross failures and bad for ranking close candidates.

The first evaluation should be designed to find failures, not to crown a winner. A 20-task pilot that exposes a schema violation, a policy breach or a tool loop is worth more than a 500-task leaderboard that confirms what you already believed.

What good reporting looks like

A decision memo can fit on one page. It should state the decision, the options, the task population and sample size, the metric and scorer, the configuration, the mean and interval for each option, the paired difference, the cost and latency, the failure mix, the limitations and the recommendation. It should say who owns the evaluation and when it will be re-run.

Three distinctions make the memo honest. First, separate verified facts from recommendations: a cited benchmark result is evidence; "use three trials" is a judgement. Second, label hypothetical numbers. Third, state what was not tested. "We did not evaluate multilingual inputs or documents longer than 40 pages" is more useful than a generic caveat.

The same discipline applies to vendor claims. Ask which harness, which subset, which date, how many trials, what scaffold, what contamination policy and whether the comparison was run independently. A benchmark number without those details is a marketing asset, not evidence.

Conclusion

Benchmarks are instruments, not verdicts. They are useful for orientation and screening, and dangerous when treated as predictions about your workload. The alternative is not a bigger benchmark; it is a smaller, owned evaluation that reflects your tasks, measures the whole system, quantifies its own uncertainty and includes cost and reliability in the decision.

Start with one decision and a documented pilot. Use its error patterns and variance to plan the larger evaluation needed for your decision; a fixed task count does not establish production fitness.

References

• Chiang, W.-L., Zheng, L., Sheng, Y., Angelopoulos, A. N., Li, T., Li, D., Zhang, H., Zhu, B., Jordan, M., Gonzalez, J. E. and Stoica, I. (2024). Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preference. Read source
• Hendrycks, D., Burns, C., Basart, S., Zou, A., Mazeika, M., Song, D. and Steinhardt, J. (2021). Measuring Massive Multitask Language Understanding. ICLR 2021. Read source
• Jimenez, C. E., Yang, J., Wettig, A., Yao, S., Pei, K., Press, O. and Narasimhan, K. (2023). SWE-bench: Can Language Models Resolve Real-World GitHub Issues? ICLR 2024. Read source
• Kapoor, S., Stroebl, B., Siegel, Z. S., Nadgir, N. and Narayanan, A. (2024). AI Agents That Matter. Read source
• Liang, P., Bommasani, R., Lee, T., Tsipras, D., Soylu, D., Yasunaga, M., Zhang, Y., Narayanan, D., Wu, Y., Kumar, A. and others (2022). Holistic Evaluation of Language Models. TMLR 2023. Read source
• Mialon, G., Fourrier, C., Swift, C., Wolf, T., LeCun, Y. and Scialom, T. (2023). GAIA: a benchmark for General AI Assistants. Read source
• Miller, E. (2024). Adding Error Bars to Evals: A Statistical Approach to Language Model Evaluations. Read source
• OpenAI (2026). Evaluation best practices. Read source
• Reuel, A., Hardy, A., Smith, C., Lamparth, M., Hardy, M. and Kochenderfer, M. J. (2024). BetterBench: Assessing AI Benchmarks, Uncovering Issues, and Establishing Best Practices. NeurIPS 2024. Read source
• Singh, S., Nan, Y., Wang, A., D'Souza, D., Kapoor, S., Üstün, A., Koyejo, S., Deng, Y., Longpre, S., Smith, N. A., Ermis, B., Fadaee, M. and Hooker, S. (2025). The Leaderboard Illusion. Read source
• Srivastava, A., Rastogi, A., Rao, A. and others (2022). Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models. TMLR 2022. Read source
• SWE-bench (2026). SWE-bench Leaderboards. Read source
• Yang, S., Chiang, W.-L., Zheng, L., Gonzalez, J. E. and Stoica, I. (2023). Rethinking Benchmark and Contamination for Language Models with Rephrased Samples. Read source
• Yao, S., Shinn, N., Razavi, P. and Narasimhan, K. (2024). τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains. Read source