When humans and model judges should score AI outputs

Some outputs have no single correct string. A support answer can be accurate but confusing; a summary can be complete but too long; a policy explanation can sound polished while inventing a rule. Human review can capture these qualities, and model judges can make some review faster. Both need a clear standard and evidence that the standard is being applied consistently.

This article explains how to build that review process, extending the cluster’s metrics and rubric guidance and the broader model benchmarking framework.

Decide what humans need to judge

Use human evaluation when quality depends on context, nuance or consequences that are hard to encode in a deterministic check. Examples include whether a response is understandable to a novice, whether it omits a material caveat, or whether an escalation is appropriate. Humans are also valuable for validating automated graders and investigating unexpected regressions.

Do not send every output to a person by default. Exact field matching, schema validity and simple policy rules may be checked automatically. A human can then review the ambiguous or high-risk cases, plus a sample of ordinary cases to monitor quality. This hybrid design uses people where their judgement adds information and preserves an audit trail.

Write a rubric with anchors

A rubric should turn vague preferences into criteria reviewers can observe. For a troubleshooting answer, separate factual correctness, relevance to the reported issue, actionability, safety and clarity. Give each score level a short anchor and one example. Define what counts as a critical failure, such as recommending a destructive command without explaining its effects.

Run a calibration round before the real evaluation. Ask reviewers to score the same small set independently, compare rationales, and discuss disagreements. If reviewers interpret “complete” differently, revise the definition. Keep the original ratings and mark adjudicated labels separately; consensus does not mean the original disagreement never existed.

Choose the number of reviewers based on risk and budget. A single reviewer may be sufficient for low-impact triage with spot checks. For a release decision, use at least two independent reviewers on a meaningful portion, and send disagreements or critical cases to an adjudicator. Report how much was double-scored and how ties were handled. Agreement statistics can summarise consistency, but they do not establish that the rubric measures the right thing.

Reduce avoidable bias

Blind model identity when practical. Remove provider names and metadata, randomise output order, and avoid showing reviewers the score they are expected to confirm. For pairwise comparisons, balance which candidate appears first and include “tie” or “both unacceptable” options. Do not always compare one polished answer with one obviously broken answer; close cases are more informative about the decision.

Reviewers can be influenced by writing style, length, formatting, familiarity with the model or the order of presentation. Position bias has been documented in studies of LLM-as-a-judge as well, where swapping candidate order can change preferences (Shi et al., 2025, version 9; first submitted 2024). Randomising order and checking reversals are simple diagnostics, not complete cures for bias.

The original MT-Bench and Chatbot Arena judge study also examines position, verbosity and self-enhancement biases in model judging. These findings motivate audits; they do not establish every human reviewer’s susceptibility to each bias.

Use model judges as instruments

A model judge can score or compare outputs at scale, but the judge has its own preferences, blind spots and failure modes. It may favour longer answers, confident tone or language patterns resembling its own. It may reward a response that cites a source without checking whether the source supports the claim. Do not treat “another model gave it 4/5” as objective ground truth.

A safer workflow is:

1. Define a narrow criterion with clear evidence requirements.
2. Provide the judge with the task, relevant source material and output, while withholding candidate identity where possible.
3. Use examples of strong, borderline and weak performance.
4. Compare judge ratings with human labels on a stratified calibration sample.
5. Inspect disagreements, especially false passes on critical failures.
6. Test order swaps, paraphrases and style changes to see whether scores move unexpectedly.
7. Revalidate after changing the judge model, prompt or task distribution.

Pairwise comparison can be easier to calibrate than absolute scoring because it asks which of two answers better meets a criterion. But it still needs a rubric, order controls and an option for ties. An absolute score can show progress against a standard, but reviewers may use scales differently. The OpenAI evaluation guide recommends specific criteria, examples and calibration of model-based judges against human feedback (evaluation best practices). This is useful practical advice, not a guarantee of agreement for your task.

Measure agreement and inspect what it means

For categorical labels, report raw agreement and a chance-adjusted statistic if appropriate. For ordinal scales, inspect the confusion matrix and how often reviewers differ by one point versus several. For pairwise rankings, compare preference rates and tie rates. Always report the sample and method: an agreement figure can change with prevalence, category balance and score definition.

Reviewers may disagree because the rubric is vague, because the task genuinely permits multiple answers, or because one reviewer lacks domain expertise. Those causes need different responses. Clarify ambiguous criteria, accept multiple valid outcomes where appropriate, or route specialist cases to qualified reviewers. Do not resolve every disagreement by majority vote if one reviewer has relevant expertise and the others do not.

As an editorial process recommendation, track reviewer burden and offer manageable review sessions. Include disclosed quality-control items and let reviewers mark uncertainty; do not assume a particular batch length causes a quantified loss of accuracy. If a reviewer’s scores drift over time, recalibrate. These are process controls, not a substitute for fair compensation and appropriate privacy safeguards.

Example: answer grounded in a policy

Imagine an assistant answering employee leave questions from an internal policy. The rubric might score: (a) whether each material claim is supported by the retrieved policy passage, (b) whether the answer addresses the question, (c) whether it states uncertainty when the text is silent, and (d) whether it avoids disclosing information outside the user’s access. The judge receives the question, policy passage and response. A human review sample checks whether the judge misses unsupported details or gives full credit to fluent but inaccurate answers.

This example is illustrative. It does not establish a measured judge accuracy rate. A real team would need policy specialists to label examples, test the judge on known failure cases and retain a manual escalation path for uncertain or high-impact outcomes.

Keep labels useful for later work

Store raw human ratings, rationales, adjudications and model-judge outputs separately. Include rubric version, reviewer role, timestamp, anonymisation procedure and relevant context. If a label changes, preserve the change history. This allows the team to distinguish model improvement from a revised definition of “good”.

Do not use a judge score as a training target or release gate without considering incentive effects. Once a model is optimised against the judge, it may learn to produce features the judge likes while missing the underlying intent. Keep a hidden audit set, inspect raw outputs and periodically compare automated scores with fresh human judgements. A mismatch should trigger investigation, not an automatic prompt tweak.

A deployment pattern

For a new feature, first use humans to define the rubric and label a small calibration set. Use deterministic checks for objective criteria, and a model judge for a limited subjective dimension only after it demonstrates acceptable agreement on relevant examples. Route missing-evidence cases, high-risk cases and judge-human disagreements to people. If you also route by a confidence score, first validate that signal against independently human-labelled errors on representative held-out cases; a judge’s self-reported confidence alone is not a validated error detector. Sample apparent passes for ongoing review. Recalibrate when model versions, prompts, source content or user populations change.

Human review is expensive and inconsistent when poorly designed; model judges scale but can repeat their own systematic errors. The goal is not to pick one judge forever. It is to create a review chain where criteria are explicit, disagreements are visible and consequential decisions have a reliable path to human scrutiny. For a robust comparison, combine this with the cluster’s guidance on repeatability, fair configurations and evaluation maintenance.