Latency, Token Usage and Cost: The Other Half of the Scorecard

A model that is three percentage points more accurate and ten times more expensive may be the wrong choice. A model that is slightly less accurate but responds in 600 milliseconds instead of four seconds may be the right one. Quality without unit economics is an incomplete evaluation, as the pillar guide argues.

This article turns token counts and list prices into a number a business can use: cost per successful task. It also explains how to measure latency in a way that reflects the user experience, and why reasoning tokens, retries and tool calls change the arithmetic.

Accuracy is not a business case

Kapoor and colleagues analysed agent benchmarks and found a narrow focus on accuracy without attention to other metrics. The result, they argued, was needlessly complex and costly systems, mistaken conclusions about the sources of accuracy gains, and a need to optimise accuracy and cost jointly (Kapoor et al., 2024). The same argument applies to model selection. A benchmark table ranks quality; it does not rank value.

The decision-relevant question is not "what does a million tokens cost?" It is "what does a successful task cost, and is that acceptable for this use case?"

Latency metrics that reflect the experience

A single average latency hides the experience. Report at least:

• Time to first token (TTFT): name the event being timed; record first visible answer content separately when reasoning or internal events precede it.
• Inter-token latency: how quickly the response streams after the first token.
• End-to-end latency: total time to a complete response.
• Throughput: requests or tokens per second under your concurrency.
• Timeout and retry rates: how often the system fails or needs a second attempt.

Report suitable latency quantiles alongside the sample size and calculation method; tail estimates such as p99 can be unstable with few observations, and an interval alone does not fix inadequate sampling (repeatability and uncertainty). A mean of 2 seconds can hide a p99 of 30 seconds, which is the experience that generates complaints. Measure from your own region and with your own concurrency; a vendor's benchmark from a different network is not your latency.

OpenAI's latency guidance makes a useful point about where time goes. Generating tokens is usually the highest-latency step: cutting output tokens by 50% may cut latency by roughly 50%. Cutting input tokens by 50% may only improve latency by 1–5% for ordinary contexts, unless the context is very large (OpenAI, latency optimization). Streaming changes perceived latency by letting the user start reading sooner.

Token accounting

Every request has several token components:

• Uncached input tokens: the prompt, context, tool definitions and retrieved documents.
• Cached input tokens: a repeated prefix that the provider can reuse, usually at a much lower price.
• Total billed output tokens: the provider-reported output category, including reasoning where its billing schema includes it. Visible answer tokens are only part of that category.
• Reasoning tokens: for OpenAI reasoning models, internal reasoning tokens are not exposed as raw reasoning text through the API, but are billed as output tokens and occupy context space (OpenAI, reasoning models).

Reasoning tokens can run from a few hundred to tens of thousands. OpenAI recommends reserving at least 25,000 tokens for reasoning and output when experimenting with reasoning models. A request that produces a 200-word answer may still bill thousands of reasoning tokens. Tool calls, retrieved content and multi-turn history add more.

The cost formula

For a completed run, first sum charges over all attempts, including failed attempts and retries:

total cost = Σ_attempts [(uncached input × input rate + cached reads × read rate + cache writes × write rate + total billed output × output rate) / 1,000,000 + tool and other selected charges]

cost per accepted task = total cost / number of accepted tasks

The rates in this expression are USD per million tokens, while tool charges are already in USD. Define acceptance before running. Count each attempt and each charge once: do not add a retry-overhead term if retries are already in the sum, and do not add reasoning again when it is included in output. If no task is accepted, report the ratio as undefined beside total cost and zero acceptances. If provider input categories overlap, first map them to mutually exclusive billed categories.

The per-attempt/success-rate calculation in the hypothetical example below is an alternative planning model, not an additional multiplier on an all-attempt total. Compare candidates under declared constraints; see configuration fairness.

Do not forget:

• judge or grader calls;
• embedding and retrieval costs;
• sandbox or code-execution charges;
• human review for escalated cases;
• engineering time to maintain prompts and tools;
• the cost of a wrong decision, which may dwarf the token bill.

Dated reference prices

Reference rates checked 26 October 2026, in USD per million tokens. These are selected API rates, not an invoice or an all-provider price comparison.

Sources: OpenAI pricing, Sol, Luna, Anthropic pricing and DeepSeek pricing. For the Sol and Luna models listed here, OpenAI prompts above 272K input tokens have 2× input/cache and 1.5× output rates for the full request. Haiku 5.5’s listed 5-minute cache-write rates are $0.125 through 100K prompt tokens and $0.625 above 100K. Other cache lifetimes, tiers, regions and tools can change charges. DeepSeek’s weekday UTC peak windows are 01:00–04:00 and 06:00–10:00, excluding Chinese public holidays; all other hours are off-peak. Recheck the current provider terms before budgeting.

The original draft’s moving Gemini promotion and unqualified Haiku row are not used as a current budget here. The preserved original remains available; this publication copy narrows the reference comparison and supplies the missing conditions. Our recorded editorial-workflow example separates observed usage from dated rate assumptions.

Hypothetical unit economics

Consider a task with 15,000 input tokens, 2,000 output tokens, 85% success and an average of 1.1 attempts per task. These numbers are illustrative, not measured.

At GPT-6 Astra list prices:

• input: 15,000 × $10 / 1,000,000 = $0.15
• output: 2,000 × $50 / 1,000,000 = $0.10
• per attempt: $0.25
• with 1.1 attempts: $0.275
• per success: $0.275 / 0.85 = $0.32

At GPT-6 Luna prices:

• input: 15,000 × $0.10 / 1,000,000 = $0.0015
• output: 2,000 × $0.50 / 1,000,000 = $0.0010
• per attempt: $0.0025
• with 1.1 attempts: $0.00275
• per success: $0.00275 / 0.85 = $0.0032

The ratio is about 100×. If the small model succeeds 70% of the time on your task, its cost per success is still about $0.0039. On these hypothetical numbers, the frontier model costs about $1.80 more for each additional success ($27.50 versus $0.275 per 100 tasks, for 15 more successes). If avoiding a failure is worth more than that, the frontier model can be the cheaper choice overall.

Now add caching. If 7,500 of the 15,000 input tokens are cached at $1.00 per million, the frontier input cost falls from $0.15 to about $0.0825, and cost per success falls to roughly $0.24. A stable prompt prefix and a stable tool schema can therefore change the economics without changing the model.

The cost-quality frontier

Plot cost per successful task on one axis and success rate on the other. The useful choices sit on the frontier: no other option is both cheaper and better. A hypothetical table:

The right choice depends on the error cost and the user experience, not on the ranking. For a low-risk internal tool, the small model may be sufficient. For a customer-facing task where an error triggers a manual correction costing $5, the frontier model may be cheaper overall.

Hidden costs

The token bill is the visible cost. The evaluation budget also includes:

• running the evaluation repeatedly across candidates;
• judge calls, which can approach the cost of the system under test;
• embeddings and vector storage for retrieval;
• sandbox compute for agents;
• human review and labelling;
• engineering time for prompts, tools and harness maintenance;
• incident response when a failure reaches a customer;
• vendor minimums, committed spend and data-residency uplifts.

A model that looks cheap per token can be expensive per successful task if it needs three retries, a longer prompt or a human check.

A measurement protocol

1. Warm up the connection and the cache.
2. Measure from the region and concurrency you will use in production.
3. Record TTFT, end-to-end latency and token counts for every request.
4. Run enough requests to report p50, p95 and p99.
5. Count retries, timeouts and parse failures.
6. Compute cost per attempt and cost per successful task.
7. Re-run after any model, prompt or price change.

The point

Cost and latency are not afterthoughts to a quality evaluation. They are part of the definition of a working system. A decision that ignores them is not a decision about production; it is a decision about a demo. The next article, on evaluating agents and tool use, shows why these numbers become even more important when the system takes multiple steps.

References

• Anthropic (2026). Plans and pricing. Read source
• Kapoor, S., Stroebl, B., Siegel, Z. S., Nadgir, N. and Narayanan, A. (2024). AI Agents That Matter. Read source
• OpenAI (2026). Latency optimization. Read source
• OpenAI (2026). Pricing. Read source
• OpenAI (2026). Reasoning models. Read source