Evaluate AI agents: outcomes, tool traces and safe boundaries

An agent can produce a convincing final message while taking the wrong action on the way. It may search the wrong source, call a tool with incorrect arguments, ignore an approval boundary or claim success after a tool failed. Evaluating only the final prose misses the behaviour that makes an agent different from a single model response.

An agent system combines a model with instructions, tools, state, control logic and an environment. AgentBench evaluates model-based agents in multiple interactive environments, illustrating why multi-step assessment differs from a static question-and-answer test. A system score belongs to that entire configuration; it should not be presented as a property of the model alone.

Define the task and authority

Consider a fictional assistant asked to prepare an expense report. It can read a sandbox ledger, look up a policy and create a draft report. It cannot submit payment or change an approval record. The correct result is a draft with supported line items and an explicit request for human review when needed. All cases and thresholds here are illustrative, not measured results.

Write the starting state, objective, available tools, permitted actions and stopping conditions for each case. Specify what counts as completion and what side effects are forbidden. Include ordinary success, ambiguous input, missing data, a tool error and a request that exceeds authority. The same final report may be acceptable in one case and unsafe in another if the agent had to take a forbidden action to obtain it.

Use a resettable sandbox or simulated tool backend for tests involving writes. Seed it with known records, isolate it from real payment and messaging systems, and verify that reset returns the same initial state. Where a simulator cannot reproduce production behaviour, record that gap. An evaluation can establish behaviour in the test environment; it does not prove safety in every real environment.

Capture a trace that supports diagnosis

For every attempt, record the case ID, system configuration, input and initial state version. Capture each tool request and response, arguments, timestamps, errors, retries, intermediate state changes and final output. Record whether a human intervened and why. Retain sensitive traces only under appropriate access and retention controls.

A trace should let a reviewer answer: what did the agent intend to do, what did the tool actually return, what state changed, and what did the agent tell the user? A tool invocation that returns an error is different from a successful state change. A final claim of completion is not evidence that the operation happened; inspect the environment's resulting state.

Separate infrastructure failures from agent failures, but do not hide either. A tool outage may be outside the model's control; the agent's response to the outage is still part of the system's behaviour. Predefine when to retry the entire case, how to count a timeout and how to report ungradable runs. The SWE-bench evaluation guide illustrates the importance of distinguishing completed evaluations, unresolved tasks and errors in a tool-based benchmark.

Score outcome and process separately

An outcome score asks whether the required state and user-facing result are correct. A process score asks whether the system stayed within authority, chose appropriate tools, handled errors and preserved evidence. Use hard failures for prohibited side effects. A draft report that is correct after an unauthorised payment attempt should not pass an authority gate.

Build checks at several levels. A schema validator can catch missing tool arguments. A state assertion can verify that no payment record changed. A reviewer can assess whether the agent explained uncertainty. A model judge may help classify traces, but calibrate it against human-labelled examples and inspect false passes. The metrics guide and judge guide cover these choices.

Include adversarial content in data the agent reads: for example, a fictional document that instructs the assistant to ignore its task and submit a payment. Treat document content as untrusted task data. The expected behaviour is to follow the system's authority boundary, not the document's embedded instruction. Also test a benign document with similar wording so a blunt refusal does not appear to solve the problem.

Control budgets and environment

Fix or disclose maximum steps, tool calls, token budget, elapsed time, retries and whether feedback from tests is available. Compare systems on the same tasks and environment when possible. If one uses a different tool suite or more attempts, report that as a system-level difference. Fair comparisons explains how to distinguish a controlled model swap from a best-system contest.

Repeat selected cases because both model choices and tool conditions may vary. Keep the environment reset between attempts and preserve case-level pairing. Record random seeds when available without claiming they guarantee complete reproducibility. Review failures that depend on race conditions, delayed tool responses or partial state changes; a successful retry may leave an earlier side effect behind.

Use component tests to find the cause

When an end-to-end case fails, replay the trace and isolate the boundary. Did retrieval omit the policy? Did the model misread a correct tool response? Did the tool reject valid arguments? Did orchestration stop too early? Substitute a fixed tool response or a controlled model output to test a hypothesis, then rerun the full system after a fix. Component success alone does not establish end-to-end success.

Report the final state, process violations, error handling, resource use and uncertainty alongside the overall completion rate. Maintain separate regression cases for any discovered failure. Before a pilot, decide which actions require human approval and what evidence is sufficient to expand scope. A narrow, monitored deployment can be appropriate while evidence accumulates; it is not a claim that the agent is generally autonomous.

The benchmarking overview places agent evaluation alongside quality, cost and risk. The maintenance guide explains how to keep these tests useful as tools and environments change.

A concrete failure-injection case

Start the fictional expense sandbox with a valid draft and configure the ledger tool to return a temporary error on the first read. The agent is allowed a bounded retry but cannot submit payment. A passing trace records the error, retries within the stated budget or asks for help, and does not claim the ledger was checked if no read succeeded. A failure includes inventing ledger entries, exceeding the retry budget or changing a payment record. Inspect both the tool log and final state; a polite final message is insufficient evidence.

Run a second case where the first read succeeds but a later write fails after partially changing the draft. Define the expected recovery before running it: for example, detect the partial state, avoid duplicating line items and tell the reviewer what remains uncertain. This exposes a class of operational failure that a clean happy-path test misses. If the sandbox cannot simulate partial writes faithfully, document that limitation and use a separate integration test in an authorised environment before relying on the result. Do not transfer sandbox actions to real accounts.

A trace for an uncertain write

This is a hypothetical evaluation trace, not a live expense operation. Score final state and authority separately; reconciliation is meaningful only where the tested tool actually supports it.

Return to the Terra cluster guide · Inspect the four-way results and evidence.