Evaluate the agent system, not only its language model

An agent can call tools, observe results, retry, delegate and decide when to stop. Its success depends on the model, but also on the instructions, tool descriptions, permissions, state, environment and orchestration code. A benchmark that scores only the final text can miss the action that caused the outcome—or credit the model for work done by the harness.

This guide focuses on tool-using systems and end-to-end task evaluation. For a broader selection process, start with the model benchmarking pillar.

Draw the system boundary

Before creating tasks, write down what the agent can access and what it cannot. Include the model identifier, system prompt, tool schemas, tool implementations, retrieval source, memory, sandbox, time limit, maximum calls, retry policy and stopping rule. Define whether a human may intervene. A benchmark result applies to that boundary; changing it changes the system being evaluated.

Distinguish a model benchmark from an agent benchmark. A model benchmark may test whether a model chooses an appropriate tool call given a prompt. An agent benchmark tests whether the complete workflow reaches an acceptable outcome through one or more calls under realistic constraints. If tool responses are mocked, the test may isolate planning but will not show how the system handles real tool errors or environmental variation. Say which level you measure.

Define task success outside the agent

A weak benchmark asks the agent to declare that it solved the task. A stronger one checks an external state or acceptance condition. For a calendar assistant, verify that the correct event exists with the right date, attendees and permissions. For code repair, run tests in a pinned environment and inspect for regressions. For a document workflow, compare required fields and confirm the source references.

Write acceptance criteria before running models. Include partial credit only when it helps diagnosis; keep a clear pass condition for the decision. Add negative tasks where the correct outcome is to ask for clarification, refuse an unauthorised action or stop because the required information is unavailable. A high completion rate is not useful if it is achieved through unsafe guesses.

Use tasks with different shapes: straightforward success, ambiguous requirements, tool failure, contradictory evidence, permission boundary, long sequence and out-of-scope request. Record task provenance and difficulty. Do not let the benchmark contain only tasks with clean descriptions and functioning tools if production is messier.

Evaluate trajectories as well as outcomes

Store each step: model input, selected tool, arguments, tool result, subsequent reasoning output if exposed, retries, handoffs, state changes and final answer. A trace can reveal whether failure came from a bad tool choice, malformed arguments, a misleading tool result, poor recovery or an incorrect stopping decision. OpenAI’s agent-evaluation guidance describes trace grading as a way to assess tool selection, handoffs and workflow-level behaviour (Evaluate agent workflows). The same principle applies outside that platform: inspect the sequence, not just the last message.

Do not overvalue a plausible trace narrative. The observable record is what happened. A model may give a convincing explanation after acting, but explanations are not proof that it followed that plan. Evaluate outcomes and action constraints directly where possible.

Control the environment

Pin repository commits, package versions, test data, permissions and network access. Reset state between attempts. If tasks rely on external APIs, save a versioned fixture or log the live dependency conditions. A task can become easier or impossible as external data changes. Give every candidate the same tool availability, rate limits, time budget and context where the question is controlled comparison.

Use sandboxing for actions that can modify files, send messages, spend money or affect accounts. Record blocked actions and whether the block was expected. Test least-privilege behaviour: an agent should not need broader access than its task requires. For high-impact operations, include an approval step in the system boundary and test whether the agent waits for it.

For coding agents, beware of test leakage and reward hacking. If hidden tests or expected patches are exposed in a repository, the benchmark may measure access to answers rather than problem solving. Conversely, weak tests may pass an incorrect patch. A recent 2026 preprint on SWE-Bench Pro Verified reports concerns about leakage and task quality in its studied benchmark version; treat that as a specific warning about benchmark construction, not as a verdict on every code benchmark (Zheng et al., 2026).

Score cost, risk and recovery

An agent that succeeds after 40 calls may be less useful than one that succeeds in 5. Record elapsed time, tokens, tool calls, retries, external charges and human interventions per task. Score harmful side effects, unnecessary actions and recoverability, not just final completion. One unsafe action may outweigh several routine successes; use hard constraints for unacceptable behaviour rather than diluting them in an average.

Measure completion, partial progress, correct abstention, recovery after tool error and side-effect quality separately. An agent that stops early may preserve safety but fail to help. An agent that perseveres may make progress but continue after the task is no longer safe. Both behaviours need context in the rubric.

Run each task more than once when trajectories are variable. Keep the initial environment identical, reset state and record random seeds or sampling settings when available. If an attempt changes persistent state, design a rollback or use isolated copies. The uncertainty guide explains how to distinguish task variation from repeated-run variation.

Example: repository issue repair

A hypothetical code benchmark can start each run from the same repository commit and issue statement. The agent receives a shell, file editor and test runner inside a network-restricted container. It has a fixed wall-clock budget and no access to the reference patch. Success requires the target tests to pass, existing relevant tests to remain green, and the diff to avoid prohibited files. The trace records commands and tool outputs. A reviewer inspects a sample of passing diffs for hard-coded tests or unrelated destructive changes.

This setup is illustrative, not a report of measured performance. It also measures a specific harness: container resources, allowed commands, test suite quality and issue selection affect the outcome. A result should name those conditions. A separate model-only test could give each candidate a single code-generation prompt and compare patches, but that would not measure repository exploration or iterative tool use.

Compare agent configurations responsibly

Use paired tasks and equal budgets when isolating model differences. If each agent uses its own planning loop or tool interface, call it a system comparison. A stronger model paired with a more capable harness may be the product choice, but attribution requires an ablation: hold the harness constant, swap the model, then hold the model constant and swap the harness where feasible.

For best-configuration comparisons, provide equal tuning time and reserve unseen tasks. Report each model’s prompt, tools, runtime, context limits, retry policy and budget. Do not quietly change instructions between candidates after observing test results. Keep task-specific tuning in development and use a fresh holdout for the final comparison.

Maintain a task bank, not just a leaderboard

Agent tasks are expensive to author and can become stale when tools and environments change. Keep a versioned task bank with acceptance tests, start-state hashes, expected permissions, reset procedure and known pitfalls. Remove or revise tasks whose success criteria are ambiguous, but preserve the previous version and explain why it changed. Track performance by task class and failure type instead of relying on one overall completion rate.

Include new incidents from real use only after privacy, consent and label review. Use redacted or synthetic variants when needed, but mark them as such. A suite of cleanly labelled tasks is easier to score; a suite that reflects actual users is more relevant. You may need both: controlled tasks for debugging and representative tasks for deployment decisions.

The key is to align the unit of evaluation with the deployed system. A model can be capable while the agent fails because its tools are poor; a reliable harness can mask model weakness on narrow tasks. Define the boundary, externally verify outcomes, preserve traces and account for cost and risk. Then your benchmark can explain not only which system succeeded, but how it failed and what to improve next. For a robust comparison, combine this with the cluster’s guidance on repeatability, fair configurations, evaluation maintenance and latency and cost.