Benchmark AI latency and cost per successful task

The cheapest token rate does not necessarily produce the cheapest completed task. A configuration may generate more tokens, retry more often or produce answers that need repair. Likewise, a fast first token does not tell you when the user receives a usable result.

Benchmark the unit of work your product promises. Record quality, elapsed time and spend together. The accounting method below is an editorial recommendation; every numerical example is hypothetical and uses invented rates rather than a current supplier quote.

Define the task clock

Choose a start event and a completion event. For a support assistant, a useful start might be when the service accepts a complete request. Completion might be when a validated answer is ready for the operator. State whether queueing, retrieval, tool waits and output validation are included.

For a streamed answer, record time to first visible content and time to complete usable output separately. A placeholder or an opening sentence may improve perceived responsiveness while leaving the substantive answer unfinished. OpenAI's latency guide discusses streaming as a way to reduce waiting for visible output; it does not make streamed completion equivalent to task completion. Latency optimisation guidance.

For an agent, completion may require a final state check after the last tool call. Do not stop the clock when it announces success if the system has not verified that the action occurred.

Apply a deadline to the task when users have one. Report successes within that deadline as well as eventual successes. An answer that arrives too late can be correct and still fail the operational requirement.

Measure under a declared load

Record concurrency, arrival pattern, region, client environment and connection reuse. A serial test with no queueing answers a different question from a service handling bursts of requests. Keep input and expected-output length distributions visible.

Report a latency distribution, such as median and 95th percentile, with the sample count and quantile method. The median describes the middle observed completion; the upper percentile helps expose slower tasks. Small samples give unstable tail estimates, so avoid presenting a highly precise percentile from a handful of calls.

Include timeout and failure rates beside successful-response latency. If the slowest requests time out and disappear from the latency calculation, a configuration can look fast because its worst experiences were excluded. State how censored or unfinished tasks are represented.

Keep a low-load diagnostic run separate from a realistic-load evaluation. The former helps investigate generation and tool behaviour; the latter supports capacity planning for the tested environment. Neither establishes unlimited production throughput.

Account for actual usage

Collect provider-reported usage categories for every attempt, plus tool calls and infrastructure costs relevant to the decision. Store raw usage fields and explain how they map to the bill. Do not infer billed usage solely from visible answer length.

Pricing can distinguish input, output, cached input and tool charges. OpenAI's current official price page lists separate categories and units; verify the applicable model, service mode and billing terms when pricing an actual run. Official pricing, accessed 22 October 2026.

Avoid double counting a category already included in another total. For example, if a reported output total includes reasoning tokens, adding a reasoning subtotal again would overstate usage. Confirm semantics with the provider's documentation before aggregating.

Track retries, failed attempts and fallback calls. A task with three model calls consumes the resources of all three, even if the final answer alone appears in the user interface. Keep development and judge costs separate from inference costs so readers know which budget the report concerns.

Make cache conditions visible

Run cold and representative warm-cache scenarios where relevant, and record the observed cache usage. Do not assume a shared prompt prefix necessarily produced a cache hit.

OpenAI's caching documentation requires matching rendered prefixes and describes model-dependent eligibility and lookup conditions. Its details can change, so a reproducible report should save the configuration and measured usage rather than assume one universal cache rule. Prompt caching documentation.

A repeated benchmark can have more prefix reuse than ordinary traffic. Conversely, a service with a large stable instruction block may benefit from reuse absent in a one-off test. State which scenario your estimate represents.

If you deliberately disable or avoid caching for a controlled experiment, keep that result separate from the deployment economics. Include cache-write charges where applicable. Report observed savings for the tested run rather than extending a best-case hit rate to all future requests.

Calculate cost per accepted task

Start with a purely hypothetical usage example. An attempt uses 2,000 input tokens at an invented $2 per million and 500 output tokens at an invented $8 per million. Its token cost is $0.004 plus $0.004, or $0.008, before tools and other charges.

Now compare complete runs, using a separate illustrative scenario. Candidate A attempts 100 tasks, spends $1.20 including retries and produces 80 accepted outcomes. Candidate B attempts the same tasks, spends $1.50 and produces 90 accepted outcomes. Their calculated cost per accepted outcome is $0.015 and approximately $0.0167 respectively.

A is cheaper on that measure; B completes more work. Neither number alone determines the decision. Check whether failures require paid human repair, whether the accepted work meets the deadline and whether either candidate violates a release gate.

Define an accepted outcome with the scoring rubric before computing the ratio. If no tasks succeed, report cost per success as undefined, alongside spend and zero successes. Do not divide by an invented minimum or omit the failed run.

Compare feasible alternatives

Apply minimum quality, safety and latency requirements before ranking cost. A low-cost configuration that invents action confirmations may be ineligible regardless of its average economics. Keep gate failures separate from tradeable preferences.

Among acceptable configurations, look for dominated choices: one that is slower, more expensive and no better on the relevant quality measures under comparable conditions. When alternatives trade strengths, state the decision's priorities instead of inventing a universal efficiency score.

Use uncertainty analysis for close comparisons and fair configuration rules for settings and attempt budgets. Cost per success is a ratio influenced by both usage and success; a single point estimate can hide variability in either.

For human-assisted workflows, optionally add a separately measured repair-cost scenario. Record the wage assumption, handling time and escalation frequency. Do not present hypothetical labour savings as realised savings.

Reconcile the run before extrapolating

Compare calculated totals with available usage records or invoices and investigate discrepancies in units, cache treatment, tools and retries. Record currency, pricing date, exchange-rate assumption if used and whether taxes are included. This series does not provide a live price comparison.

Publish attempted tasks, accepted tasks, total spend, cost per accepted task, latency distribution, deadlines, cache conditions and unresolved accounting gaps together. For stateful work, use the agent guide to define completion. The benchmarking guide then connects operating economics to a choice supported by actual task outcomes.