Measure AI latency and cost per successful task

A model’s headline token price is not the same as the cost of a useful result. The system may read a long prompt, generate multiple drafts, call tools, retry failures or send difficult outputs to a human. It may also be fast on average but too slow for the users who need it. Add operational measures to quality evaluation before choosing a model.

This guide explains how to measure latency and cost in a workload-specific benchmark. It complements fair model comparisons and the benchmark pillar.

Define what a completed task means

Choose a unit that matches the product: one answered question, one processed document, one resolved ticket or one successful agent run. Define success before calculating cost. A malformed answer that needs a human correction is not a successful completion just because the API call returned HTTP 200. A workflow that correctly abstains and routes to a specialist may be successful if that is the intended outcome.

Track the path to completion. A request may involve one model call or several; record input and output tokens, cached tokens if billed differently, tool invocations, retries, timeouts, and human-review minutes if those are part of your economic decision. Report gross cost per attempt and cost per successful task. If success rate is low, the latter can be much higher.

Calculate cost transparently

A simple estimate is:

total workload cost = model charges across all attempts + tool charges across all attempts + human-review cost

Model charges include every applicable input and output charge, including retries. Tool charges likewise include repeated calls. Do not add a separate retry line when those charges are already included. Keep categories mutually exclusive; add other actually applicable costs explicitly.

Divide by the number of successful tasks, not merely the number of initiated requests. This formula is a bookkeeping aid, not a universal billing model. Providers may charge by tokens, calls, audio duration, image input, compute time or other units. Check the current official price schedule and terms for the endpoint you use. OpenAI’s published pricing, for example, distinguishes token categories and lists separate prices for some tool calls (official pricing). Those rates are time-sensitive and vary by model and service options; this article intentionally does not reproduce a fixed price table.

Record the date, region, currency, service tier, caching assumptions and whether taxes or platform charges are included. Measure token usage on representative inputs. Two models can tokenise the same text differently, and one may generate longer answers. If requests use large repeated instructions, caching may affect the economics; verify actual billing records rather than assuming cache eligibility.

For a batch of 1,000 tasks, calculate at least three scenarios: ordinary traffic, a difficult-input mix and a stress or peak-traffic case. Keep assumptions visible, such as average input size, expected retry rate and percentage sent to review. A spreadsheet with hidden assumptions is less useful than a plain table showing them.

Measure latency users feel

Latency needs a start and end definition. For a streaming interface, time to first token may matter, but the first token can be an unhelpful preamble. Measure time to first useful output as well as time to completion. For a back-office workflow, total end-to-end time including retrieval, tools, parsing and retries may be the relevant measure.

Test under representative concurrency and geography. Include cold and warm conditions if they differ, and state how you handled warm-up. Report median and upper-tail percentiles such as p95, alongside timeout and error rates. The average can conceal a slow tail that causes users to abandon a request. Preserve the time distribution rather than publishing only a single number.

If providers stream at different speeds or return different amounts of text, compare at the product level: when can the user act, and when is the result complete enough to trust? Keep quality constant in the interpretation. A short, fast answer may omit required facts; it is not a win if it fails the task rubric.

Include retries and fallbacks

A retry may improve completion rate and increase cost and latency. Count each attempt and mark why it occurred. Distinguish a retry caused by your application from one caused by a provider error. If the product falls back to another model, include the fallback path in the system benchmark and report how often it is used.

Avoid excluding slow or failed requests because they look like outliers. Exclude only when the evaluation harness itself was broken under a predeclared rule. A provider outage is not necessarily a model-quality defect, but it is operational evidence relevant to service selection. Report it separately rather than erasing it.

Compare on matched workload and quality

Use the same cases, expected outputs and workload settings for each candidate. If one system reaches a quality threshold with more calls, its cost per successful task should reflect them. Report a quality-cost curve where feasible: cost at several operating points can reveal whether a cheap model only works with substantial review or retries.

Do not compare a price quote for one configuration to measured usage from another. Estimate using the observed token and tool counts from the benchmark, then apply the dated price schedule. Include a sensitivity analysis for uncertain variables like review time, traffic volume and retry rate. Cost assumptions are often less stable than the model’s measured output on a fixed test set.

Example worksheet

The following example is hypothetical and shows the arithmetic structure, not measured provider performance. A team processes 500 support cases. Candidate A makes one call per case and succeeds on 440; its total API and tool charges are $18, and 60 cases need 4 minutes each of human review. Candidate B costs $27 in API charges, succeeds on 475 and sends 25 cases to 4-minute review. To compare cost per successful case, the team must value reviewer time and define whether corrected cases count as successful. If review time costs $1 per minute and corrections meet the service contract, A’s total is $258 and B’s is $127, yielding different economics than API charges alone suggest. The values are invented for illustration and should not be reused as benchmark results.

Assuming every reviewed case is successfully repaired, both workflows complete 500 cases: A costs $258 / 500 = $0.516 per assisted completion; B costs $127 / 500 = $0.254. Their autonomous pass rates remain 88% (440 / 500) and 95% (475 / 500), respectively. Do not label assisted completions autonomous successes. If repairs fail, reduce the successful-completion denominator and report those failures. With zero successful outcomes, cost per success is undefined; report the spend and zero successes rather than a finite ratio.

The same worksheet should include median and p95 end-to-end latency, output quality and critical failures. A low total cost cannot compensate for an unacceptable safety failure. Apply hard quality gates first, then compare economics among candidates that pass.

Preserve a decision-ready record

Include the workload profile, number of tasks, quality definition, token and call counts, retry and review policies, timestamped prices, latency measurement boundaries and environment. Report both per-attempt and per-successful-task values. Keep the assumptions alongside the results so another person can recompute a forecast when rates or traffic change.

Finally, revisit the calculation after launch. Input length, user behaviour, prompt changes and model updates can shift consumption. Pricing can change independently of quality. Re-run the workload evaluation when a relevant system component changes, and keep historical cost figures dated. A decision made from yesterday’s token price and today’s traffic estimate is not a timeless property of the model.