Choose LLM evaluation metrics and write a rubric that works

A score is useful only if you can explain what earns it. “Quality: 8 out of 10” leaves too much room for a reviewer to reward fluency, a parser to reward formatting or a team to average away a consequential error.

Build scoring from the task's requirements. Use objective checks for observable conditions and anchored rubrics for judgements that need context. Keep critical failures visible. The worked examples below are hypothetical calculations and editorial recommendations, not measured benchmark findings.

Map each requirement to evidence

Start with the task contract. For a support assistant, the requirements might be correct policy interpretation, an appropriate next step, truthful action status and a readable explanation. Those are four claims, so do not initially hide them in one score.

Choose the strongest practical check for each. A schema validator can establish that required fields exist. A lookup can verify an order identifier. A reference-backed reviewer can assess whether a return condition was omitted. A state check can establish whether an authorised action actually occurred.

Keep syntax and semantics separate. A well-formed object can contain an incorrect eligibility decision. A response can name the right tool but use the wrong account. Passing a format check should not be reported as task success.

For code or calculations, execute meaningful checks in a controlled environment where appropriate. Document the inputs they cover. A finite test suite provides evidence for its cases, not proof of correctness for every possible input.

Define the denominator before the score

Write how every scheduled task enters the report. If the assistant times out, returns an empty output or violates the required format, does it count as failure? For a user-facing success measure, the usual recommended default is to retain those tasks in the denominator and report the failure types separately.

Distinguish harness defects from system failures. A broken reference file may make a case unscorable. A model response that cannot be parsed may be exactly the failure the task was designed to detect. Do not discard both under the label “invalid”.

State exclusions with counts and reasons. If you remove cases after seeing results, explain the change and apply it consistently to every candidate. Preserve the earlier calculation so a reviewer can inspect the effect.

Also define abstention. Asking for an absent delivery date may pass a missing-information task while failing a task where the date was supplied. The denominator and reference behaviour must reflect those different conditions.

Work through a classification example

Suppose an illustrative routing test contains 100 requests, 20 of which require escalation. A candidate flags 25 requests, correctly including 15 of the 20 escalation cases. It has 15 true positives, 10 false positives and five false negatives.

Precision for escalation is 15 divided by 25, or 60%. Recall is 15 divided by 20, or 75%. The F1 score is twice the product of precision and recall divided by their sum, approximately 66.7%. These definitions follow the standard precision, recall and F-score formulation. Scikit-learn metric reference.

There are also 70 true negatives, making accuracy 85%. Always predicting “no escalation” would achieve 80% accuracy on this constructed set while missing every escalation. That arithmetic explains why accuracy alone is insufficient for this decision.

Choose the emphasis from consequences. If missed escalations are particularly costly, inspect recall and the false-negative cases. If unnecessary escalations overwhelm operators, inspect precision and workload. F1 is a summary, not an automatic statement that both error types have acceptable costs.

For multiple categories, report class-level results and explain averaging. A macro average gives classes equal weight; a traffic-weighted summary answers a different operational question. Keep the class counts alongside either measure. Averaging definitions.

Write an anchored rubric for open answers

For the hypothetical support answer, use a narrow rubric for policy correctness. A score of zero means the answer contradicts a material supplied condition or invents eligibility. A score of one means the central answer is correct but omits a condition needed for the next step. A score of two means the answer correctly handles all material conditions relevant to the case.

Define other dimensions separately. For next-step quality, full credit might require either actionable instructions supported by the policy or a necessary clarification. For unsupported claims, record whether the response asserts facts or actions absent from the evidence.

Give graders examples near boundaries. Show a concise correct answer, a fluent answer that omits a crucial condition and an answer that honestly asks for missing information. Explain the reference rationale. The goal is consistent interpretation, not imitation of one preferred wording.

Avoid a rubric that rewards length automatically. “Comprehensive” can encourage irrelevant detail unless you specify which information is required. Prefer “includes the applicable return window and the necessary next step” to “gives a thorough response”.

Separate release gates from preferences

Some errors should not be compensated by strengths elsewhere. An invented refund confirmation should not disappear inside a high readability average. Define a critical-failure flag and report its numerator, denominator and examples.

Then define task success as an explicit combination. For illustration: correct material policy conditions, appropriate next step and no unsupported action claim. Readability can be reported separately or included only if a defined readability defect prevents use.

If you create a weighted composite, disclose weights and show the underlying dimensions. Check whether reasonable alternative weights change the decision. Do not choose weights after learning which candidate they favour.

The gate threshold is an operational decision. A dataset with no observed critical failures does not establish zero real-world risk; it establishes a result on the tested cases. Use uncertainty analysis to describe the strength of that evidence.

Audit the scorer as part of the system

Test the scorer on intentionally wrong, incomplete and malformed answers as well as correct ones. Include a plausible hallucination, a correct paraphrase, a justified refusal and an output containing irrelevant instructions to the grader. Check which failures the scorer misses.

If a model judge assesses open-ended answers, compare its labels with independent human decisions and inspect disagreements. A judge that agrees on easy passes may still miss the critical failures you care about. The grading process guide shows how to calibrate and monitor it.

Version the rubric, parser and reference evidence together. Re-score stored outputs when a scoring defect is corrected, and identify the affected results. Do not present a score change caused by a new grader as a model improvement.

Finally, put each metric beside the decision it informs: critical failures for release eligibility, task success for usefulness, error slices for remediation and cost per successful task for economics. That report fits into the benchmarking decision guide without asking one number to carry incompatible meanings.