Public Benchmarks Are a Screening Tool, Not a Decision

A vendor sends a slide with three benchmark scores and a rank. A model you had not considered is suddenly at the top. The temptation is to treat the rank as a recommendation. It is not. A public benchmark tells you how a system performed on a standardised test. Whether that performance transfers to your invoice extraction, your clinical intake or your code review workflow is a separate question, and it is the question that matters.

The useful distinction is between screening and selection. Screening is cheap, broad and removes obviously unsuitable options. Selection is expensive, narrow and predicts performance in your environment. Public benchmarks are excellent screening tools and weak selection tools.

What a public benchmark actually is

Every benchmark is a bundle of four choices: a task sample, a harness, a metric and a date. The sample defines the population, the harness defines prompts, tools, retries and scoring, the metric defines success, and the date defines the world the test was built for.

MMLU contains 57 tasks spanning elementary mathematics, history, computer science, law and other subjects. Its authors found that even the largest models of the time lagged expert accuracy on every task (Hendrycks et al., 2021). BIG-bench took a different approach: 204 tasks from 450 authors across 132 institutions, with human raters as a baseline. It found that performance and calibration improve with scale but remain poor in absolute terms, and that some apparent "breakthrough" behaviours depend on brittle metrics (Srivastava et al., 2022).

HELM measured seven metrics — accuracy, calibration, robustness, fairness, bias, toxicity and efficiency — across 16 core scenarios and 26 targeted scenarios for 30 models under standardised conditions. It found that models had previously been evaluated on only 17.9% of core scenarios on average, and raised that coverage to 96.0% (Liang et al., 2022). The lesson is not that HELM solved evaluation. It is that a single accuracy number hides the trade-offs that determine whether a system is usable.

Construct validity: is this test measuring the skill?

The first question to ask of any benchmark is whether it measures the skill you care about. This is construct validity, and it is routinely skipped.

A multiple-choice knowledge test measures recall and elimination under a constrained format. It does not measure whether a model can follow a 12-field output schema, cite a retrieved passage, refuse an out-of-policy request, or recover when a tool returns malformed JSON. A code-generation benchmark that runs unit tests measures functional correctness on small, self-contained problems; it does not measure whether an agent can navigate a large repository, run the test suite, interpret a failure and try again.

Public benchmarks are also vulnerable to saturation. When frontier models cluster near the ceiling, the differences between them shrink toward the measurement noise, and the ranking becomes unstable. A two-point gap on a saturated test may reflect prompt formatting rather than capability.

Leaderboards are measurements with politics

Leaderboards compress complex evidence into a rank. That compression is useful, and it is also where distortion enters.

Chatbot Arena measures pairwise human preference. It is a genuine signal: at the time of its paper it had more than 240,000 votes, and the authors reported good agreement between crowd and expert raters (Chiang et al., 2024). But preference is not task fitness. A model can be pleasant to talk to and still be unreliable at extracting line items from a scanned invoice.

The ranking process can also be shaped by the participants. An analysis of Arena data documented private testing before public release, selective disclosure of results, and large asymmetries in how often models are sampled. It estimated that two large providers received about 19.2% and 20.4% of all arena data, while a combined 83 open-weight models received about 29.7%, and it found that access to arena data could produce substantial performance gains on the arena distribution (Singh et al., 2025). Those findings do not make the leaderboard worthless, but a rank is evidence about a moving process with unequal access, not a verdict about general capability.

A separate assessment of 24 benchmarks against 46 lifecycle best practices found large quality differences and that most benchmarks did not report statistical significance or allow easy replication (Reuel et al., 2024). If the benchmark itself does not report uncertainty, a rank difference of a few places may carry no information.

What public benchmarks are good for

Used correctly, public benchmarks do four jobs well.

Shortlisting. They reduce a long list of models and providers to two to four plausible candidates. A model that fails badly on a relevant public test is unlikely to be the right choice, even if your task differs.

Regression against known capability. If a new model release claims a large gain on a well-understood benchmark, you can check whether the claim is plausible and whether the gain comes from the capability you need.

Sanity-checking a vendor claim. Ask which subset, which harness, which date and which trial count. A vendor number that cannot be reproduced under a documented configuration is a marketing asset, not evidence.

Learning the vocabulary. Benchmark names are a useful map of capability areas: knowledge, reasoning, coding, tool use, long-context retrieval, multilingual performance. They help a team ask better questions.

What they cannot do

They cannot estimate performance on your distribution, price your task, validate your tools or retrieval, certify reliability across runs, or tell you whether a failure will be merely annoying or commercially serious.

A benchmark reports an average. Your workload has a particular mix of easy, rare and dangerous cases. A model that is excellent on average and catastrophic on the rare case may look good on a leaderboard and be unacceptable in production.

A two-stage selection rule

I recommend a simple two-stage process.

Stage 1: screen with public evidence. Choose two or three public benchmarks that are genuinely relevant to the capability you need. Record the benchmark version, subset, harness and date. Add availability, region, context window, data-handling terms, list price and latency. Reduce the field to two to four candidates. Do not spend weeks optimising a public score.

Stage 2: select with your own tasks. Build a private pilot of 50–150 tasks drawn from your workload. Include the current baseline. Run every candidate on the same tasks with the same scorer and the same configuration. Measure success, cost, latency and the failure mix. Report an interval, not just a mean.

The pilot need not be perfect, only representative enough to expose a gross mismatch. A 60-task pilot that reveals a schema failure or a policy breach has already earned its cost.

A shortlist scorecard

A scorecard keeps the two kinds of evidence separate:

Failure case: the leaderboard winner that could not follow instructions

Consider a hypothetical team building a document-intake pipeline. They screen five models on a general knowledge benchmark and a coding benchmark, then choose the highest scorer. In production, the model produces valid JSON on about 70% of documents. The failures are schema failures: commentary before the JSON object and inconsistent handling of null fields.

A 100-document pilot with a schema validator would have caught this in a day. The benchmarks were not wrong; they measured a different skill. The team confused screening with selection.

Interrogating a benchmark claim

Before you rely on any published number, ask:

1. Which benchmark version and subset?
2. Which harness and scaffold?
3. Which prompt format, and was it the same for every model?
4. How many trials, and what is the reported uncertainty?
5. What date was the run, and what model snapshot?
6. What contamination controls were used?
7. Who ran it, and did they have a commercial interest in the result?

If several answers are "not reported", treat the number as a hint, not a decision. For the next step, see designing evaluation tasks and datasets that reflect your workload; for the leakage risks that can inflate a public score, see contamination, leakage and holdouts. The broader process is set out in the pillar guide, and the system-level version for agents is in evaluating agents and tool use.

The rule to remember

Public benchmarks answer how a system did on a standard test. They do not answer whether it will work for you. Use them to earn a place in the pilot, then decide with your own tasks.

References

• Chiang, W.-L., Zheng, L., Sheng, Y., Angelopoulos, A. N., Li, T., Li, D., Zhang, H., Zhu, B., Jordan, M., Gonzalez, J. E. and Stoica, I. (2024). Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preference. Read source
• Hendrycks, D., Burns, C., Basart, S., Zou, A., Mazeika, M., Song, D. and Steinhardt, J. (2021). Measuring Massive Multitask Language Understanding. ICLR 2021. Read source
• Liang, P., Bommasani, R., Lee, T., Tsipras, D., Soylu, D., Yasunaga, M., Zhang, Y., Narayanan, D., Wu, Y., Kumar, A. and others (2022). Holistic Evaluation of Language Models. TMLR 2023. Read source
• Reuel, A., Hardy, A., Smith, C., Lamparth, M., Hardy, M. and Kochenderfer, M. J. (2024). BetterBench: Assessing AI Benchmarks, Uncovering Issues, and Establishing Best Practices. NeurIPS 2024. Read source
• Singh, S., Nan, Y., Wang, A., D'Souza, D., Kapoor, S., Üstün, A., Koyejo, S., Deng, Y., Longpre, S., Smith, N. A., Ermis, B., Fadaee, M. and Hooker, S. (2025). The Leaderboard Illusion. Read source
• Srivastava, A., Rastogi, A., Rao, A. and others (2022). Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models. TMLR 2022. Read source