Measure AI latency, token use and cost alongside quality

A model that produces excellent answers but misses a response-time target or requires expensive retries may not be the best choice for a particular workflow. Conversely, the cheapest request is not necessarily the cheapest successful task. Evaluate quality, speed and cost together, using the same configuration and workload that the decision concerns.

The following calculations describe a fictional policy-answering assistant. All rates, volumes and timings are hypothetical, not provider prices or measured results. Actual prices, token categories, availability and terms change; consult the relevant provider's current official pricing and contract on the date of your own decision.

Define the user-visible clock

Record when a request enters the application, when the first useful output reaches the user and when the complete answer is available. The first byte is not always useful: a progress marker may appear before an answer. For an agent, also record total task completion time, tool waits and any human approval delay. State which events start and stop each clock.

Measure end to end as well as component times. Retrieval, network transit, queueing, model generation, tool execution and rendering can all contribute. A faster model call may not improve the user experience if a slow retrieval service dominates the path. Instrument the components, but make the final service-level decision on the path users actually experience.

Report median and a high percentile such as p95 with the number of observations, distribution of input sizes, concurrency, region, time window and timeout rate. The median describes a typical observation in the measured set; a high percentile helps reveal a slow tail. A p95 from a tiny sample is unstable. Do not omit timed-out requests from the narrative simply because they have no completion time. Report them separately and define how they affect the service objective.

Count all billable work

Capture input and output tokens per attempt using provider usage records where available. Note separately reported categories such as cached input or reasoning tokens when the provider exposes and bills them differently. Avoid assuming that displayed output text accounts for every billed token. Record tool calls, retrieval infrastructure, retries, orchestration, storage and human review when they materially affect total cost.

For an illustrative calculation, suppose one attempt uses 2,000 input tokens and 400 output tokens. Assume input costs $2 per million tokens and output costs $8 per million. The model charge is (2,000 × 2 + 400 × 8) / 1,000,000 = $0.0072 per attempt. At 10,000 attempts, that component is $72. These invented rates exclude taxes, infrastructure, tools, discounts and review; they must not be treated as a price quote.

If 8,000 of those 10,000 attempts meet a predeclared quality standard, the illustrative model charge per accepted success is $72 / 8,000 = $0.009. This ratio does not mean a user should pay that amount or that all failures are equally costly. Include all attempts and the full cost of retries in the numerator. State whether a successful task required one or several attempts and how failures and human corrections were handled.

Make workload assumptions visible

Token use depends on context length, retrieved passages, conversation history and output length. Report distributions rather than only a single average. Include long-context and unusually verbose cases if they matter to your workload. A few large inputs can dominate a budget; truncation can reduce cost while harming quality, so test the effect rather than assuming it is free.

Estimate monthly spend using a range of volumes and observed or carefully stated token distributions. Separate fixed infrastructure from per-request costs. Include cache hit assumptions only if your workload can achieve them, and model retry or fallback rates explicitly. For self-hosted systems, include hardware, utilisation, operations and capacity headroom rather than comparing only a marginal electricity figure with an API bill.

The denominator should match the business question. Cost per request helps capacity planning. Cost per accepted answer includes quality and retry effects. Cost per completed user task can also include human review, tool operations and later correction. Keep the quality definition stable across candidates; a lower cost per success calculated with a looser rubric is not a fair comparison.

Test under realistic load

Use the same case mix and concurrency plan for each candidate where feasible. Measure warm and cold conditions if both occur in production, and document caching, rate limits, region and network path. Run enough observations to describe variation. If a provider throttles requests, record the resulting wait or error rather than silently dropping it.

Do not extrapolate from an idle single request to peak traffic. A system can meet its median target at low load and fail its tail target at peak concurrency. Run a staged load test with appropriate limits and authorisation, observe timeouts and error rates, and avoid assuming that a provider's rate limits or performance will remain unchanged. Recheck before a material deployment decision.

Make the trade-off explicit

Create a scorecard with quality gates, critical failure rate, latency distribution, total cost and the assumptions behind each number. A candidate that fails a critical safety criterion should not win solely on speed or cost. Among candidates that meet the gates, compare the additional quality gained for additional spend and delay. If results depend on an uncertain price or volume, show a sensitivity range.

Keep the run manifest and raw usage evidence with the evaluation, subject to privacy controls. Recalculate when pricing or workload changes without rewriting the original record. For configuration parity, see fair comparisons; for interpreting uncertain quality differences, see repeatability.

Diagnose a slow or expensive tail

Inspect individual traces near the slow end of the distribution. Was the delay in queueing, retrieval, generation, a tool call or a retry? Group by input length and task type, and distinguish a consistently slow case from a transient service event. An aggregate percentile alone cannot identify which component to fix. If streaming is used, compare time to first useful content with total completion time; a quick opening followed by a long stall may still frustrate users.

For cost, inspect the highest-token cases and identify whether long context is necessary. A shorter retrieval set might lower cost but omit the very exception the answer needs. Test the revised retrieval on the same quality and critical-failure cases before adopting it. Likewise, a smaller output limit may truncate an answer or its citation. Record the quality, latency and cost of each configuration together. If a saving depends on an assumed cache hit rate or discount, show what happens when that assumption is lower; do not treat a favourable estimate as an observed bill.

Keep usage categories from being counted twice

OpenAI’s reasoning guide explains that reasoning is charged within output tokens. Provider reporting and rates can differ; retain missing fields as unavailable. The earlier $0.0072 example assumes one input rate for all 2,000 tokens and 400 total billed output tokens. It has no cache discount and no extra reasoning charge. The table is illustrative, not this benchmark’s recorded usage or a current price quote.

Return to the Terra cluster guide · Inspect the four-way results and evidence.