Evaluating Agents and Tool Use: Model Benchmarks Are Not System Benchmarks

An agent is not a model with a longer prompt. It is a system that observes, plans, calls tools, reads results and decides what to do next. The model is one component. The scaffold, the tool definitions, the memory, the retrieval, the sandbox, the policy layer and the user simulator are others. Change any of them and the system's behaviour changes.

That is why a model benchmark cannot tell you whether an agent will work. A model can score well on a knowledge test and still loop on a simple API call, call the right tool with the wrong arguments, or give up after one error. Evaluating an agent means evaluating the whole loop, including its reliability across repeated runs, as the pillar guide sets out.

The model is one component

A useful decomposition:

• Model: the underlying language or multimodal model.
• Scaffold: the loop that decides when to plan, call a tool, reflect or stop.
• Tools: the functions available, their descriptions, schemas and error behaviour.
• Memory and state: conversation history, scratchpads, persistent stores.
• Retrieval: what context is fetched and when.
• Policy: permissions, guardrails, approval gates and refusal rules.
• Environment: the sandbox, browser, filesystem, database or simulated user.
• Runtime: timeouts, retries, concurrency and cost caps.

A change in the scaffold can move the score more than a change in the model. Kapoor and colleagues argue that the needs of model developers and downstream developers have been conflated, making it hard to identify which agent is best for a particular application; they also criticise inadequate holdouts and a lack of standardisation (Kapoor et al., 2024). The practical response is to evaluate the system you will deploy, with the scaffold you will use.

What public agent benchmarks measure

Public agent benchmarks are useful for screening, but each measures a particular system.

SWE-bench contains 2,294 real GitHub issues from 12 Python repositories. The original paper found that the best model of the time resolved only 1.96% of issues (Jimenez et al., 2023). The leaderboard now reports several subsets: a 500-instance Verified set filtered by humans, a 300-instance Lite set, a 300-instance Multilingual set across 42 repositories and nine languages, and a Bash Only view that runs every model in the same mini-SWE-agent environment (SWE-bench leaderboards). That last detail matters: the scaffold is part of the measurement.

τ-bench emulates a conversation between a simulated user and an agent with domain-specific API tools and policy guidelines. It scores the final database state against an annotated goal state and introduced pass^k to measure reliability. Even strong function-calling agents succeeded on fewer than 50% of tasks, and pass^8 was below 25% in the retail domain (Yao et al., 2024).

GAIA contains 466 real-world questions requiring reasoning, multimodality, browsing and tool use. Humans answered 92%; GPT-4 with plugins answered 15%. The benchmark holds back answers to 300 questions to power a leaderboard (Mialon et al., 2023). The gap is a useful reminder that tool use and multi-step execution are capabilities in their own right.

LiveCodeBench continuously collects new coding problems, enabling evaluation on a selected post-cutoff window where the model’s training cutoff is known. Such a window reduces, but does not eliminate, contamination risk (Jain et al., 2024).

These benchmarks are valuable. None of them is your agent.

Outcome oracles

Start with the end state. What would prove the task was completed?

• Final answer: a value, a file, a report.
• Executable tests: the repository's test suite passes.
• Database state: the order is cancelled, the refund is issued, the record is updated.
• File diff: the patch changes the intended lines and nothing else.
• Policy compliance: no prohibited action occurred.

An executable oracle is better than a judge when the task has a checkable end state (objective metrics and rubrics). For a customer-support agent, the database state after the conversation is often a stronger signal than the transcript. For a coding agent, the tests are the oracle, with the caveat that they only test what they test.

Trajectory metrics

The outcome tells you whether the agent succeeded. The trajectory tells you how, and why it failed.

Useful metrics:

• Tool selection accuracy: did it choose the right tool?
• Argument accuracy: did it pass the right values?
• Steps and tool calls: how many, and how many were redundant?
• Recovery rate: after an error, did it recover?
• Handoff accuracy: in a multi-agent system, did it route to the right agent?
• Policy violations: did it attempt a prohibited action?
• Cost and latency: tokens, tool charges and elapsed time (latency, token usage and cost).

OpenAI's agent evaluation guidance recommends starting with traces when debugging behaviour, then moving to datasets and eval runs for repeatability (OpenAI, agent workflows). Trace grading is useful because the final answer can be right for the wrong reasons.

Reliability across runs

A single success establishes that attempt’s outcome, rather than repeated-run reliability. Report pass^k: the proportion of tasks where all k attempts succeed. Also report the distribution of outcomes across runs, the variance in steps and cost, and tail latency.

A hypothetical example: an agent has 72% pass@1 on a 150-task set, but only 38% pass^5. That means it succeeds on most tasks sometimes, and on a minority of tasks consistently. For unattended workflows requiring consistent repeated success, pass^5 adds important evidence; assess both measures against the actual execution and retry policy. The cost per successful task is also higher than the single-attempt cost suggests, because failures consume tokens and tool calls.

Designing a private agent evaluation

A workable design:

1. Define the task and the end state. Write the success oracle before building the environment. A harness such as Inspect supports composable datasets, agents, tools, scorers and sandboxes, which helps keep those pieces explicit (UK AI Security Institute, Inspect).
2. Reset the environment for every run. A dirty database or a cached file makes runs incomparable.
3. Stub the tools deterministically. Real APIs introduce variance you did not intend to study. Use fixtures, then validate against a small live sample.
4. Constrain the user simulator. If a model plays the user, fix its script or constrain its behaviour, and evaluate the simulator separately.
5. Cap steps, time and cost. Record what happens when the cap is hit; a timeout is a failure mode, not a missing data point.
6. Capture the trace needed for diagnosis: tool calls, arguments, results, model messages and errors. Prefer synthetic fixtures; redact credentials and unnecessary personal or confidential data, restrict access, and set a retention period. Record any resulting evidence gaps.
7. Hold out tasks. Do not tune the scaffold on the evaluation set.
8. Run multiple trials. Report pass^k, not just pass@1.
9. Include prohibited actions. Safety is part of the evaluation, not a separate review.
10. Record the configuration. Model, scaffold, tools, prompts, sandbox and date.

Ablation

To attribute a gain, change one thing at a time:

• same scaffold, different model;
• same model, different scaffold;
• with and without a tool;
• with and without retrieval;
• with a stricter or looser policy.

Ablation is how you avoid the conclusion that "the new model is better" when the real change was a better tool description or a retry loop. It is also how you decide whether to pay for a more expensive model: if the scaffold accounts for most of the gain, the model may not be the lever.

A failure taxonomy

Classify failures so the next iteration has a target:

• Wrong tool: the right action was available but not chosen.
• Wrong arguments: the right tool was called with incorrect values.
• Loop: repeated calls without progress.
• Premature stop: the agent declares success before the end state is reached.
• Policy violation: a prohibited action was attempted or completed.
• Unrecoverable error: the agent cannot proceed after a tool failure.
• Handoff failure: the request is routed to the wrong agent or never returned.
• Cost blowout: the task succeeds but at an unacceptable cost or latency.

A failure taxonomy turns a single success rate into a diagnosis. It also makes comparisons between models more honest, because different models fail in different ways.

Safety and permissions

An agent that can act needs least privilege, approval gates for irreversible actions, sandboxing, audit logs and a kill switch. Evaluation should include prohibited actions, not just task success: attempts to access another customer's record, send an external email, delete a file or bypass an approval. A system that completes tasks but violates policy is not a successful system.

Reporting

Report success with an interval, pass^k, steps, cost, latency percentiles, the failure mix and the configuration manifest. State which parts are model, scaffold, tools and environment. A reader should be able to tell whether the result applies to their stack.

The point

Agent evaluation is system evaluation. The model matters, but so does everything around it. Measure the end state, inspect the trajectory, test reliability across runs, and treat safety as part of the outcome. The final article, on interpreting results and maintaining evaluations, turns these measurements into a decision that stays useful.

References

• Jain, N., Han, K., Gu, A., Li, W.-D., Yan, F., Zhang, T., Wang, S., Solar-Lezama, A., Sen, K. and Stoica, I. (2024). LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code. Read source
• Jimenez, C. E., Yang, J., Wettig, A., Yao, S., Pei, K., Press, O. and Narasimhan, K. (2023). SWE-bench: Can Language Models Resolve Real-World GitHub Issues? ICLR 2024. Read source
• Kapoor, S., Stroebl, B., Siegel, Z. S., Nadgir, N. and Narayanan, A. (2024). AI Agents That Matter. Read source
• Mialon, G., Fourrier, C., Swift, C., Wolf, T., LeCun, Y. and Scialom, T. (2023). GAIA: a benchmark for General AI Assistants. Read source
• OpenAI (2026). Evaluate agent workflows. Read source
• SWE-bench (2026). SWE-bench Leaderboards. Read source
• UK AI Security Institute (2024). Inspect: Framework for Large Language Model Evaluations. Read source
• Yao, S., Shinn, N., Razavi, P. and Narasimhan, K. (2024). τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains. Read source