Objective Metrics and Rubrics That Measure What Matters

Every evaluation makes a claim about success. The metric is how that claim becomes a number. Choose the wrong metric and you can improve the score while making the product worse: a summariser that scores well on word overlap but invents facts, a classifier that is accurate on average but fails the one category that matters, a code model that passes easy tests and cannot repair a failing build.

This article maps common decisions to metric families, explains what each family does and does not measure, and gives a method for writing rubrics that two reviewers can apply consistently. It assumes you already have a task set with an oracle or a reference answer, as described in designing evaluation tasks and datasets and the pillar guide.

The metric is part of the claim

Start with the decision and the error cost. "Choose a model for support replies where a policy breach is unacceptable and a slightly verbose answer is fine" implies a pass/fail gate on policy, then a graded score for helpfulness. "Choose a model for code review where a missed security issue is worse than a false positive" implies recall on security findings before precision or style.

Write the success criterion before looking at outputs. If you look first, you will gravitate to a metric that makes the outputs you have seen look good. That is not measurement; it is rationalisation.

Metric families

Exact match and normalised match. Good for classification labels, extracted identifiers, dates, amounts and structured fields. Normalise only distinctions the task contract declares irrelevant; preserve case-sensitive identifiers, meaningful punctuation and numeric precision. Exact match is harsh on legitimate paraphrase, so use it where the format is part of the contract.

Token-level F1. Useful for extractive question answering and short answers where partial credit is meaningful. It rewards overlap, not correctness: an answer can contain the right tokens in a misleading order and still score well.

pass@k. Measures whether at least one of k sampled attempts succeeds. The standard unbiased estimator, from the HumanEval work, is 1 - C(n-c, k) / C(n, k), where n ≥ k samples are drawn and c are correct; compute it per task, then average across tasks (Chen et al., 2021). It answers a capability question: can the model ever solve this? It does not answer a reliability question.

pass^k. Measures whether all k attempts succeed. It is the metric for a system that must work every time, not just once. τ-bench introduced it for agent reliability and found that strong function-calling agents could succeed on fewer than half of tasks and be markedly less consistent across eight trials (Yao et al., 2024). Use pass@k for capability and pass^k for dependability.

Functional correctness. Run the code, execute the query, validate the schema, compare the database state. This is the strongest oracle when the task has a checkable end state. SWE-bench scores a model by whether its patch resolves the issue under the repository's tests (Jimenez et al., 2023). The limitation is coverage: the tests only check what they check.

Retrieval metrics. Recall at k, precision at k, mean reciprocal rank and answer groundedness. Use them when the system's job is to find evidence, not just to produce prose.

Tool selection and argument accuracy. For agents, score whether the right tool was called with the right arguments, separately from whether the final answer was correct. OpenAI's evaluation guidance calls these tool selection and data precision, and treats them as distinct from functional correctness (OpenAI, evaluation best practices). For agent-specific metrics, see evaluating agents and tool use.

Cost and latency. Not quality metrics, but part of the decision. A model that is 2 percentage points better at twice the cost may be the wrong choice.

What each family misses

Exact match punishes a correct answer expressed differently. Token F1 trades precision against recall and rewards overlap; extra unmatched tokens can lower it. This differs from a model judge’s possible verbosity bias. BLEU and ROUGE were designed for translation and summarisation. Using them as a proxy for open-ended quality is an anti-pattern; they are useful for regression against a stable reference, not as a verdict on "good" (OpenAI, evaluation best practices). pass@k hides inconsistency. pass^k hides capability. A test suite hides the cases it does not test.

There is no universal metric. There is only a metric that is appropriate to a decision, with known failure modes.

Rubric design

When the output is open-ended, a rubric is a structured judgement. A good rubric has five properties:

1. Observable criteria. "Cites the policy section" is observable. "Is helpful" is not.
2. A small number of levels. Three to five, with a clear description of each.
3. Anchored examples. Show what a 1, 3 and 5 look like. OpenAI's guidance recommends a "show rather than tell" policy with examples at different score levels (OpenAI, evaluation best practices).
4. Binary gates before graded dimensions. Policy compliance, factual accuracy and safety should pass or fail before tone or style is scored. This prevents a polished but unsafe answer from scoring well.
5. No double-barrelled criteria. "Accurate and concise" is two criteria. Split them, or one will hide the other.

A worked rubric for a support reply:

The gates come first. A reply that fails policy compliance is not rescued by excellent tone.

Aggregation

Do not collapse everything into one number unless the decision requires it. Report per-criterion and per-stratum results, then an aggregate with an explicit weighting. A system that scores 85% overall but 40% on the high-risk stratum is not an 85% system for a high-risk use case.

Macro-averaging (average of stratum scores) answers "how does it do across categories?" Pooling task outcomes describes performance across the sampled tasks. It estimates traffic-level performance only when the sample reflects traffic proportions, or when appropriate population weights are applied. Report sampling proportions and weights. They can differ substantially when strata have different sizes. State which one you used.

For a pass/fail decision, define the threshold in advance. "Ship if the lower bound of the 95% interval on the high-risk stratum is above 90%" is a decision rule. "Ship if it looks good" is not.

Scorer validation

A scorer is an instrument, and instruments drift. Validate it:

• Agreement with humans. Label 50–100 outputs by hand and compare. Report agreement and inspect disagreements. This is especially important for LLM judges.
• False-positive and false-negative audits. Read the cases the scorer marked as successes and failures. You will find patterns the aggregate hides.
• Adversarial cases. Include outputs that are fluent but wrong, correct but terse, and correct but formatted differently. Check that the scorer handles them as intended.
• Stability. Run the scorer twice on the same outputs. If it is stochastic, measure the variation.
• Versioning. Change the rubric or the judge and you have a new instrument. Record the version with every score.

Failure cases

Metric gaming. A model learns to produce the format the scorer rewards, not the outcome the user needs. This is especially likely with regex-based scoring and LLM judges that reward structure.

Verbosity bias. Longer answers can receive inflated scores from model judges; overlap metrics have their own precision/recall behaviour (Zheng et al., 2023). Control for length or score concision explicitly.

Formatting rewards. A JSON output that is valid but semantically empty passes a schema check. Pair schema validation with field-level correctness.

Hard-case exclusion. Dropping cases the scorer cannot handle makes the evaluation easier and the result less useful. Keep them and report them as a separate category.

A rubric no one shares. If two reviewers interpret "completeness" differently, the score measures the reviewers. Pilot the rubric, measure agreement, revise.

A metric selection checklist

1. What decision will this number inform?
2. What is the cost of a false positive and a false negative?
3. Is there an executable oracle? If so, use it.
4. If not, what rubric criteria are observable and gated?
5. Does the metric reward length, format or confidence rather than correctness?
6. Have you validated the scorer against human labels?
7. Are you reporting per-stratum results and an interval?
8. Is the metric versioned with the dataset and the judge?

The point

A metric is a compressed argument about what matters. Make the argument explicit, test the instrument, and report where it is weak. The next article, on human evaluation and model judges, takes the hardest case: when the scorer is another person or another model.

References

• Chen, M., Tworek, J., Jun, H. and others (2021). Evaluating Large Language Models Trained on Code. Read source
• Jimenez, C. E., Yang, J., Wettig, A., Yao, S., Pei, K., Press, O. and Narasimhan, K. (2023). SWE-bench: Can Language Models Resolve Real-World GitHub Issues? ICLR 2024. Read source
• OpenAI (2026). Evaluation best practices. Read source
• Yao, S., Shinn, N., Razavi, P. and Narasimhan, K. (2024). τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains. Read source
• Zheng, L., Chiang, W.-L., Sheng, Y., Zhuang, S., Wu, Z., Zhuang, Y., Lin, Z., Li, Z., Li, D., Xing, E. P., Zhang, H., Gonzalez, J. E. and Stoica, I. (2023). Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena. NeurIPS 2023. Read source