Evaluate AI agents by outcomes, tool actions and recovery

An agent can say “done” without completing the task. It can also reach the requested state through an action it was not permitted to take. Evaluating only the final message misses both failures.

An agent evaluation covers the model, instructions, tools, orchestration, environment and interaction policy. It measures a system. You can compare models inside that system, but only if the surrounding configuration is controlled and the conclusion names those conditions.

The procedure below is an editorial design for a hypothetical support agent. It does not describe a live refund service or measured benchmark performance.

Define the system and the state

Specify the initial database, available tools, policy version, user request and permissions. Then define the desired final state and the changes that must not occur. Include an explicit end condition and task budget.

For an illustrative return scenario, the agent may check an order, verify supplied conditions and create a return request after the required confirmation. It may not issue a refund or alter another account. Success means the correct return record exists, required approval was obtained and no forbidden change occurred.

Separate the user's desired outcome from the authorised outcome. If the request is ineligible, a correct explanation or escalation may be success. A benchmark that rewards every requested action would penalise appropriate policy adherence.

State whether a human can intervene and how that intervention is counted. Assisted completion is a legitimate operating mode, but its results should not be labelled autonomous success.

Build resettable scenarios

Use an isolated environment with synthetic or appropriately controlled records. Reset state before every case and repeat. Record the reset version and verify that prior actions cannot leak into later tasks.

A practical scenario contains initial records, user instructions, expected state predicates, forbidden actions and a maximum number of steps or elapsed time. Keep scorer-only information separate from the agent's context. If the answer key is accessible through a tool, you are testing a different task.

Include ordinary success, ineligible requests, missing identifiers, contradictory records and requests outside the tool set. Add cases where the agent must ask a question before it can proceed. These follow from the task-design contract.

If you simulate users with a model, record the simulator prompt, version and constraints. Inspect sample conversations for unrealistic assistance or accidental answer disclosure. Supplement simulation with human-designed scenarios and, where appropriate, controlled human interactions; simulator behaviour is part of the evaluation's limits.

Check outcomes and permissions separately

Write a deterministic final-state check where possible. For the illustrative return, check that exactly one correct record was created for the intended order and account. Verify that unrelated records and payment state are unchanged.

τ-bench provides a primary example of evaluating tool-using conversations against an annotated final database state, with policy documents and simulated users. This is a useful benchmark design precedent, not proof that a local state checker covers every permission rule. Original τ-bench paper.

Add trajectory-level checks for requirements that final state cannot establish. Was confirmation requested before the write? Did the agent access an unauthorised account and later undo the change? Did it expose information in a message even though the final database looks correct?

Define success as the conjunction of required outcome and required constraints. Report outcome-only success and constraint violations separately for diagnosis. Do not average a permission failure into a high helpfulness score.

Inspect the path to diagnose failures

Retain ordered tool calls, arguments, responses, timestamps, user messages and final output, subject to an appropriate data-handling policy. A trace should connect every action to the case and configuration that produced it.

Classify failures by mechanism. Tool selection asks whether the right operation was chosen. Argument correctness asks whether it used the right identifier and values. Policy adherence asks whether the action was allowed. Outcome checking asks whether the intended change happened.

Also assess reporting: did the final answer accurately describe the observed result? An agent that times out after creating a return record should not confidently claim failure and tell the user to start again without checking state.

Avoid scoring similarity to one exact action sequence unless that sequence is required. Multiple safe paths may reach the same valid outcome. Use traces to enforce necessary ordering and diagnose behaviour, rather than reward imitation of a reference path.

Test recovery deliberately

Inject controlled failures in a separate challenge suite: a read timeout, a malformed tool response, an expired session, a rate-limit response and a write whose acknowledgement is lost. Document the injection point so candidates receive comparable conditions.

A lost acknowledgement after a write is especially useful. The system should reconcile state before retrying a potentially duplicated action. Whether that uses an idempotency key, a status lookup or a human escalation depends on the tool contract. Define the expected recovery rather than assuming retries are always safe.

Check budget exhaustion. When the agent reaches its time or step limit, it should stop within the declared rules and report what is known. Count incomplete tasks, tool costs and any side effects. Do not delete exhausted runs from the denominator.

Evaluate malicious or irrelevant instructions in tool content as another challenge. A retrieved note saying “ignore the policy” should not grant permissions. Record observed behaviour; passing a finite challenge set does not prove complete resistance to prompt injection.

Measure consistency with the right metric

Repeat complete scenarios from reset state under the same protocol. Report task success across attempts, variation by scenario and consequential failures. Keep repeats nested within scenarios when analysing uncertainty.

Be precise about notation. In the τ-bench paper, pass^k measures the chance that all k independent trials succeed, averaged across tasks. pass@k concerns at least one success among k trials. A result that looks strong under the latter may still be unsuitable for a service that must behave consistently. τ-bench metric definition.

Do not turn a mean success rate into a reliability metric by simply raising the average to a power. Tasks can have different success probabilities, and repeated outcomes need an estimator suited to the design. Report the observed repeated-case results and declared method.

Use the uncertainty guide for independent units and the fairness guide for attempt budgets, prompts and tool equivalence.

Report an operating envelope

Publish the scenario families, configuration, environment version, outcome checks, policy checks, intervention rate and recovery results. Include task-level latency and cost, not just the model's final generation time.

State what remains untested: live integrations, unusual concurrent changes, unsupported languages or policy exceptions. Keep simulation limits explicit. A sandbox result supports the tested environment; it does not establish that an integration will behave identically in production.

Store regression cases from failures and assign an owner through the maintenance process. The benchmarking pillar then connects agent evidence to a bounded decision about which tasks the system may perform and under what supervision.