Benchmark AI latency and cost per successful task

The cheapest token rate does not necessarily produce the cheapest completed task. A configuration may generate more tokens, retry more often or produce answers that need repair. Likewise, a fast first token does not tell you when the user receives a usable result.

Benchmark the unit of work your product promises. Record quality, elapsed time and spend together. The accounting method below is an editorial recommendation; the original numerical examples are hypothetical and use invented rates. The dated editorial-benchmark case below uses recorded usage and explicit reference rates.

Define the task clock

Choose a start event and a completion event. For a support assistant, a useful start might be when the service accepts a complete request. Completion might be when a validated answer is ready for the operator. State whether queueing, retrieval, tool waits and output validation are included.

For a streamed answer, record time to first visible content and time to complete usable output separately. A placeholder or an opening sentence may improve perceived responsiveness while leaving the substantive answer unfinished. OpenAI's latency guide discusses streaming as a way to reduce waiting for visible output; it does not make streamed completion equivalent to task completion. Latency optimisation guidance.

For an agent, completion may require a final state check after the last tool call. Do not stop the clock when it announces success if the system has not verified that the action occurred.

Apply a deadline to the task when users have one. Report successes within that deadline as well as eventual successes. An answer that arrives too late can be correct and still fail the operational requirement.

Measure under a declared load

Record concurrency, arrival pattern, region, client environment and connection reuse. A serial test with no queueing answers a different question from a service handling bursts of requests. Keep input and expected-output length distributions visible.

Report a latency distribution, such as median and 95th percentile, with the sample count and quantile method. The median describes the middle observed completion; the upper percentile helps expose slower tasks. Small samples give unstable tail estimates, so avoid presenting a highly precise percentile from a handful of calls.

Include timeout and failure rates beside successful-response latency. If the slowest requests time out and disappear from the latency calculation, a configuration can look fast because its worst experiences were excluded. State how censored or unfinished tasks are represented.

Keep a low-load diagnostic run separate from a realistic-load evaluation. The former helps investigate generation and tool behaviour; the latter supports capacity planning for the tested environment. Neither establishes unlimited production throughput.

Account for actual usage

Collect provider-reported usage categories for every attempt, plus tool calls and infrastructure costs relevant to the decision. Store raw usage fields and explain how they map to the bill. Do not infer billed usage solely from visible answer length.

Pricing can distinguish input, output, cached input and tool charges. OpenAI's current official price page lists separate categories and units; verify the applicable model, service mode and billing terms when pricing an actual run. Official pricing, accessed 22 October 2026.

Avoid double counting a category already included in another total. For example, if a reported output total includes reasoning tokens, adding a reasoning subtotal again would overstate usage. Confirm semantics with the provider's documentation before aggregating.

Track retries, failed attempts and fallback calls. A task with three model calls consumes the resources of all three, even if the final answer alone appears in the user interface. Keep development and judge costs separate from inference costs so readers know which budget the report concerns.

Make cache conditions visible

Run cold and representative warm-cache scenarios where relevant, and record the observed cache usage. Do not assume a shared prompt prefix necessarily produced a cache hit.

OpenAI's caching documentation requires matching rendered prefixes and describes model-dependent eligibility and lookup conditions. Its details can change, so a reproducible report should save the configuration and measured usage rather than assume one universal cache rule. Prompt caching documentation.

A repeated benchmark can have more prefix reuse than ordinary traffic. Conversely, a service with a large stable instruction block may benefit from reuse absent in a one-off test. State which scenario your estimate represents.

If you deliberately disable or avoid caching for a controlled experiment, keep that result separate from the deployment economics. Include cache-write charges where applicable. Report observed savings for the tested run rather than extending a best-case hit rate to all future requests.

Calculate cost per accepted task

Start with a purely hypothetical usage example. An attempt uses 2,000 input tokens at an invented $2 per million and 500 output tokens at an invented $8 per million. Its token cost is $0.004 plus $0.004, or $0.008, before tools and other charges.

Now compare complete runs, using a separate illustrative scenario. Candidate A attempts 100 tasks, spends $1.20 including retries and produces 80 accepted outcomes. Candidate B attempts the same tasks, spends $1.50 and produces 90 accepted outcomes. Their calculated cost per accepted outcome is $0.015 and approximately $0.0167 respectively.

A is cheaper on that measure; B completes more work. Neither number alone determines the decision. Check whether failures require paid human repair, whether the accepted work meets the deadline and whether either candidate violates a release gate.

Define an accepted outcome with the scoring rubric before computing the ratio. If no tasks succeed, report cost per success as undefined, alongside spend and zero successes. Do not divide by an invented minimum or omit the failed run.

Compare feasible alternatives

Apply minimum quality, safety and latency requirements before ranking cost. A low-cost configuration that invents action confirmations may be ineligible regardless of its average economics. Keep gate failures separate from tradeable preferences.

Among acceptable configurations, look for dominated choices: one that is slower, more expensive and no better on the relevant quality measures under comparable conditions. When alternatives trade strengths, state the decision's priorities instead of inventing a universal efficiency score.

Use uncertainty analysis for close comparisons and fair configuration rules for settings and attempt budgets. Cost per success is a ratio influenced by both usage and success; a single point estimate can hide variability in either.

For human-assisted workflows, optionally add a separately measured repair-cost scenario. Record the wage assumption, handling time and escalation frequency. Do not present hypothetical labour savings as realised savings.

Reconcile the run before extrapolating

Compare calculated totals with available usage records or invoices and investigate discrepancies in units, cache treatment, tools and retries. Record currency, pricing date, exchange-rate assumption if used and whether taxes are included. The worked case below is a dated reference calculation; it does not provide a continuously updated price comparison.

Publish attempted tasks, accepted tasks, total spend, cost per accepted task, latency distribution, deadlines, cache conditions and unresolved accounting gaps together. For stateful work, use the agent guide to define completion. The benchmarking guide then connects operating economics to a choice supported by actual task outcomes.

Recorded case · our three-model editorial benchmark

Our editorial benchmark extension offers an observed usage example, separate from the hypothetical examples above. The counters are runtime-reported; the dollar amounts below are estimates using rates checked 26 October 2026, in USD per million tokens.

Cached input + uncached input + total billed output equals total tokens. Cache share uses input as its denominator. Reasoning is already included in output; DeepSeek’s separate reasoning and unpersisted verifier/background usage are unavailable, not zero. Native tokenizers and harnesses differ.

Download: SVG · PNG · CSV · Link to figure · Full-precision chart data

View chart data (display rounded)

Recorded production tokens and cache use

Producer	Cached input	Uncached input	Total billed output	Total tokens	Cached share of input
GPT-6.1 Sol	2,894,720	152,100	41,141	3,087,961	95.01%
GPT-6 Luna	3,009,152	101,268	47,354	3,157,774	96.74%
DeepSeek V4.1 Flash	31,187,584	690,792	177,989	32,056,365	97.83%

Sources: Frozen scores · Run identities · Reference costs.

For these records, cache writes were zero. Calculate each request as (uncached input × input rate + cached input × cache-read rate + total billed output × output rate) / 1,000,000, then sum the requests. Include every attempt once. Output includes any reasoning subtotal already reported within it; adding reasoning again would double count. An unreported reasoning field is not evidence of zero reasoning usage.

The reference input/cache-read/output rates are $2/$0.10/$10 for Sol, $0.10/$0.01/$0.50 for Luna, and $0.15/$0.003/$0.60 for DeepSeek. Sources: Sol model pricing, Luna model pricing and DeepSeek pricing. These are OpenAI Standard and DeepSeek direct-API off-peak reference rates, without tools or subscription allocation. The CommandCode run occurred on Sunday 25 October UTC, outside DeepSeek’s weekday peak windows. This does not attest CommandCode’s invoice or its served snapshot.

Sol’s arithmetic is (152,100 × 2 + 2,894,720 × 0.10 + 41,141 × 10) / 1,000,000 = $1.005082. Cached input is a subset of total input, so charging all input at the full rate and then adding cached input would be wrong.

For these Sol and Luna model designations, OpenAI prompts above 272,000 input tokens apply 2× input/cache rates and 1.5× output rates to the entire request. None of these frozen Sol/Luna producer requests crossed that threshold. The new judge did: 26 of 65 requests qualified. Applying the rule per request yields $3.9670558 for that review’s 14,735,836 total tokens. Comparing the accumulated session total with a per-request threshold would misprice it.

The request-level export and calculation reproduce these totals. Producer cache-hit counts are recorded data, not assumed hit percentages. The fresh review is separate overhead; the original two-way review, setup/preflight, this publication goal, tools, taxes, unavailable background/verifier usage and subscription charges are excluded from these production rows. No invoice, editor-accepted task denominator or actual editing-time saving was measured. Consequently these figures are token-only reference estimates, not cost per successfully published article or realised customer savings.

Four-way production usage · 28 October 2026

The new four-model editorial report preserves the historical three-way estimates above and adds frozen Terra usage. Cached input is inside input; reasoning is inside output. Terra used 791,539 input tokens (728,192 cached) and 24,021 output (601 reasoning), totalling 815,560. The separate native goal meter of 79,625 is not cumulative request usage. Missing fields remain unavailable.

Cached input + uncached input + total output equals total production tokens. Cached input is included in input and reasoning in output. Cache share uses total input. DeepSeek reasoning is unavailable, not zero. Native tokenizers/reporting differ; judge overhead is excluded. No measured task charges or current comparable API-equivalent costs are available.

Download: SVG · PNG · CSV · Link to figure · Full-precision chart data

View chart data (display rounded)

Four-way recorded production tokens and cache use

Producer	Total input	Cached input (included)	Uncached input	Total output	Reasoning output (included)	Total input + output	Cache share of input
GPT 6.1 Sol	3,046,820	2,894,720	152,100	41,141	5,500	3,087,961	95.01%
GPT 6.1 Terra	791,539	728,192	63,347	24,021	601	815,560	92.00%
GPT 6 Luna	3,110,420	3,009,152	101,268	47,354	9,353	3,157,774	96.74%
DeepSeek V4.1 Flash	31,878,376	31,187,584	690,792	177,989	Unavailable	32,056,365	97.83%

Sources: Frozen scores · Run identities.

Terra’s 348.139854-second workflow includes tools. DeepSeek’s 3,518.355932 seconds includes permission waits. These observations are not inference-speed measurements. Tokenizers, dates, harnesses and supervision differ. The quality judge’s 18,467,115 tokens and 2,523.402688 seconds are separate overhead. No task invoice was measured and no comparable current rate set was verified for this four-way round; cost remains unavailable. See the frozen usage and qualifications.