Human reviewers and model judges: how to evaluate nuanced AI answers

Many useful AI outputs have no single exact answer. A policy explanation may be accurate but incomplete, a summary may omit a consequential exception, and a refusal may be appropriate only when the supplied evidence is insufficient. Human review can assess these qualities. A model judge can help scale the work, but it must be evaluated as a measurement system in its own right.

Consider a fictional assistant that answers questions from supplied policy excerpts. The examples here are hypothetical; they are not results from an experiment. A good assessment asks whether each material claim is supported, whether a relevant exception is included and whether the citation supports the claim. Fluency alone is not enough.

Design the human review first

Give reviewers the question, the evidence available to the candidate and a concise rubric. If reviewers see information the system never received, they may penalise it for not knowing something unavailable. Conversely, if a reference is needed to assess correctness, supply it to reviewers without leaking it into the candidate's input. Record which evidence each party saw.

Remove candidate identity where practical. Randomise output order and keep formatting comparable, while retaining content needed to judge the task. Ask reviewers to label independently before discussing disagreements. Provide a way to mark a case as ambiguous or ungradable, with a reason. Forcing every case into pass or fail can conceal defects in the question or reference.

A rubric should distinguish factual support, completeness, citation and appropriate uncertainty. Include examples of each boundary and a rule for the overall decision. For example, a correct conclusion with an unrelated citation can fail the citation criterion even if the conclusion is supported elsewhere. The rubric guide shows how to write those criteria.

Train reviewers with a small calibration set, then compare their independent decisions. Examine disagreement by category and case, not just a single agreement percentage. Reviewers may share a misconception; agreement alone does not establish truth. A domain expert can adjudicate hard cases, but retain original labels and document the rationale. If the rubric changes, version it and consider relabelling a consistent sample.

Decide what the model judge should do

A judge can classify an answer against a criterion, compare two answers, or identify supporting passages. Narrow tasks are easier to inspect than a single open-ended request to decide which model is best. Provide the candidate output, relevant evidence and rubric; request a structured label and a short evidence-based explanation that reviewers can audit. Do not treat an explanation as proof that the label is correct.

For a pairwise comparison, show both candidates under the same conditions and randomise their order. Include a tie or “both fail” option where appropriate. A polished but unsupported answer should not win simply because its alternative is terse. Judging the Judges reports position bias in pairwise LLM judging across the authors' tested judges and tasks. Test order effects locally by swapping responses and measuring how often the preference changes. A swap test detects some instability; it does not eliminate every bias.

Do not assume that a judge is neutral because it is a different model from the candidate. It may prefer a familiar style, verbosity, or its own family's outputs, and it can misunderstand the evidence. Candidate text is also untrusted input: a response may contain instructions aimed at the grader. Delimit the material being judged, instruct the judge to treat it as data, and include adversarial examples in calibration. No prompt alone guarantees resistance.

Calibrate against independently labelled cases

Build a reference set that includes easy passes, subtle omissions, incorrect citations, appropriate abstentions, over-refusals and plausible but unsupported claims. Have qualified humans label it independently of judge outputs. Split calibration examples from a separate validation set; tuning the judge prompt against every labelled example leaves no clean estimate of generalisation.

Compare the judge with adjudicated labels using a confusion matrix, especially false passes on critical failures. Overall agreement can be misleading if almost all cases are easy passes. Inspect errors by slice and by candidate, because a judge may favour one style or fail on a particular document type. If the validation set is small, report counts and uncertainty rather than claiming precise reliability.

Set an acceptable error policy before scaling. A team might use automatic passing only for low-risk cases, route uncertain or critical cases to humans, and periodically sample automatic decisions for audit. The threshold is a local policy choice, not a universal number. If the judge cannot reliably identify a critical error, keep that criterion under human review or redesign the task.

Run and maintain the process

Version the judge model, prompt, rubric, examples and settings. Retain the input, output, label and reason where privacy rules allow. Track human review cost and turnaround alongside judge cost. A cheaper judge can still be expensive if it sends many cases to adjudication or allows errors that require later remediation.

Revalidate after a model alias changes, a rubric is revised, the candidate output style changes or new case types appear. A judge that works on short answers may struggle with long documents. Monitor agreement and false-pass rates over time with a fresh human-labelled sample. Do not add every disagreement to the judge's prompt and then report performance on those same examples as independent validation.

Human judgement and model judging are complementary. Humans provide domain context and can discover missing criteria; automation can apply a validated rule at greater scale. Both need clear evidence, versioned instructions and checks for error. Connect the resulting labels to uncertainty analysis before treating small differences as meaningful, and to the overall benchmarking guide before making a deployment decision.

Budget for disagreement

Plan for the time needed to resolve difficult cases, not merely the time to label easy ones. A practical review sheet can capture the independent labels, the supporting passage each reviewer selected, the point of disagreement and the adjudicator's decision. If disagreement repeatedly concerns the same exception, the underlying policy may need clarification. Mark those cases as ambiguous until the policy owner resolves the issue rather than forcing reviewers to guess.

For a model judge, sample both automatic passes and automatic failures for human audit. Reviewing only low-confidence cases can miss confident false passes. Track how the sample was selected and retain enough examples from each candidate and important slice to find uneven performance. If the judge and humans disagree, inspect the evidence before changing the label or the prompt. A judge's confident explanation is useful for locating its error, but it does not replace independent evidence. Put the cost and capacity of this audit in the evaluation plan so that scaling automation does not silently eliminate oversight.

A filled adjudication record

This illustrates a review record; it does not report actual human labels or establish that the four-way judge was calibrated against humans.

Return to the Terra cluster guide · Inspect the four-way results and evidence.