Human Evaluation and Model Judges: Getting Reliable Judgements at Scale

Some outputs have a single correct answer: a date, a total, a passing test. Many do not. A support reply can be accurate and cold, complete and confusing, policy-compliant and unhelpful. To evaluate those outputs you need a judgement, and a judgement comes from a person or a model acting as one.

Both are instruments. Both have reliability, bias and cost. The mistake is to treat either as ground truth. A human panel can disagree with itself; a model judge can prefer longer answers, its own family's style or the option presented first. This article is about designing the judgement so that it measures quality rather than the quirks of the judge, within the decision loop the pillar guide sets out.

When humans are necessary

Use human evaluation when:

• the decision is high-stakes or hard to reverse;
• quality is subjective or context-dependent;
• policy nuance matters;
• the volume is small enough to afford it;
• you need to calibrate or validate an automated judge.

Do not use humans for high-volume regression where an executable oracle exists. A test suite is cheaper, faster and more consistent. Save human attention for the cases where judgement is the point.

Human study designs

Pairwise preference. Show two outputs and ask which is better for a stated criterion. Pairwise judgements are usually more reliable than absolute scores, because people are better at comparing than calibrating. Randomise which output appears first.

Rubric grading. Ask reviewers to score each output against explicit criteria. Useful for tracking absolute quality and for diagnosing which dimension failed. Requires a well-anchored rubric.

Error annotation. Ask reviewers to mark specific defects: factual error, policy breach, missing step, unclear instruction. Good for building a failure taxonomy and for training an automated scorer.

Think-aloud review. Ask reviewers to narrate their reasoning on a small sample. Expensive, but excellent for discovering criteria you did not know you had. Shankar and colleagues describe "criteria drift": reviewers need criteria to grade outputs, but grading outputs helps them discover the criteria, and some criteria depend on the outputs observed (Shankar et al., 2024). Build in a revision round.

Whichever design you choose, blind the model identity, randomise order, have reviewers work independently, and record disagreements before discussion.

Writing the rubric for humans

The rubric rules from the previous article apply, with extra emphasis on interpretation:

• define each criterion in observable terms;
• give an example for each level;
• include a "cannot judge" option;
• state what to do when two criteria conflict;
• run a 20-item pilot and revise the guide.

A rubric that works for a model judge may be too terse for people, and vice versa. Test it with the actual reviewers.

Agreement: why percent agreement misleads

If two reviewers agree on 80% of cases, that sounds good until you notice that 70% of cases are easy passes. Most of the agreement is explained by the base rate. Chance-corrected statistics account for this.

Cohen's kappa for two raters and nominal categories:

kappa = (p_o - p_e) / (1 - p_e)

where p_o is observed agreement and p_e is agreement expected by chance. A kappa of 0 means chance-level agreement; 1 means perfect agreement. Interpret the number relative to the task and the consequence of error: 0.7 may be acceptable for a low-stakes style judgement and inadequate for a safety gate.

Krippendorff's alpha for more than two raters, or for ordinal data. It generalises the same idea and is a better choice when you have a panel or a graded scale.

Report the statistic, the number of items and the number of raters. A kappa computed on 20 items is not precise. If agreement is low, do not simply tell reviewers to try harder; find the ambiguous criterion and fix it.

Model judges

A model judge can grade at a scale and cost that humans cannot match. It can also be wrong in systematic ways.

The MT-Bench and Chatbot Arena work found that strong judges such as GPT-4 could match both controlled and crowdsourced human preferences over 80% of the time, roughly the level of agreement between humans. It also documented position bias, verbosity bias, self-enhancement bias and limited reasoning ability, and proposed mitigations (Zheng et al., 2023). Those findings are the right starting point: promising, but conditional.

Common judge patterns:

• Pairwise comparison. Ask which of two outputs is better on a stated criterion. More reliable than absolute scoring, but sensitive to order. OpenAI's guidance recommends pairwise comparison or a pass/fail gate for reliability (OpenAI, evaluation best practices).
• Single-answer grading. Ask for a score against a rubric. Cheaper, but noisier and prone to scale drift.
• Reference-guided grading. Give the judge a reference answer or a checklist. Usually the most reliable for factual tasks.

Mitigations that work

Swap the order. Before aggregation, encode A-win as +1, tie as 0 and B-win as −1 by candidate identity, not screen position. Score both orders and retain both votes. Averaging neutralises a pure first-position preference in this encoding, but does not prove removal of other biases. Predeclare whether discordant votes become a tie, an inconclusive result or an adjudication. The original MT-Bench paper also describes a conservative rule requiring wins in both orders.

Hide model identity. Remove names, styles and metadata that let the judge infer the source. The original judge study’s self-enhancement evidence was inconclusive; model-family/style preference remains a risk to assess rather than an established explanation for every result.

Control for length. Either instruct the judge to ignore length or score concision separately. Check whether longer answers receive higher scores despite no improvement in the target criterion.

Give a reference. A reference answer or a checklist anchors the judgement and reduces drift.

Require evidence before the score. Ask for a brief justification grounded in the task evidence, then a score. Test whether this procedure improves agreement on a held-out calibration set; account for its tokens and cost.

Use pass/fail gates. For policy and factual accuracy, a binary gate is more stable than a 1–10 scale.

Use more than one judge. If two judges disagree systematically, that is information. Average where appropriate, and report the disagreement.

Cap the judge's cost. A judge that costs as much as the system under test may still be worth it, but count it in the evaluation budget.

A validation protocol

Before trusting a judge, validate it:

1. Sample 50–100 items stratified by difficulty and by the quality dimension you care about.
2. Have humans label them blind, using the same rubric the judge will use.
3. Run the judge blind, with order randomisation and no model identifiers.
4. Compute agreement overall and per stratum. A judge can be good on average and bad on the dimension that matters.
5. Read the disagreements. Are they judge errors, human errors or rubric ambiguity? Each has a different fix.
6. Revise and re-check on a fresh sample. Do not tune the judge on the same items you use to report agreement.
7. Freeze a calibration set and re-run it whenever the judge model or rubric changes.

A useful target is agreement comparable to human–human agreement on the same task. That is not a universal pass mark, but it is a defensible benchmark. Report the residual disagreement; do not round it away.

Judge drift

A judge is a model, and models are updated, deprecated and replaced. A judge that agreed with your experts in March may not agree in September. Version the judge, keep the calibration set, and re-validate after any change; maintaining evaluations treats this as routine. If you cannot pin the judge version, treat the comparison across time as suspect.

The same applies to human reviewers. New hires, new guidelines and new product behaviour change the panel. Re-run calibration periodically.

Failure cases

The judge that loves length. It rewards padding. Control length or score concision.

The judge that recognises its own family. It favours outputs that sound like itself. Blind the source and use multiple judges.

The rubric no one shares. Low agreement is treated as a training problem rather than a design problem.

The panel that agrees because they discussed it. Independence is lost; the agreement statistic is meaningless.

The judge tuned on the test. The calibration set and the evaluation set overlap. The judge looks accurate and generalises poorly.

The single judge on a high-stakes decision. No cross-check, no fallback, no human review.

The point

Human judgement and model judgement are both measurements, and measurements need calibration. Use humans where judgement is the task, use model judges where scale is the constraint, and validate both against a shared rubric. The next article, on repeatability and uncertainty, shows how to put an interval around the agreement and the scores.

References

• OpenAI (2026). Evaluation best practices. Read source
• Shankar, S., Zamfirescu-Pereira, J. D., Hartmann, B., Parameswaran, A. G. and Arawjo, I. (2024). Who Validates the Validators? Aligning LLM-Assisted Evaluation of LLM Outputs with Human Preferences. Read source
• Zheng, L., Chiang, W.-L., Sheng, Y., Zhuang, S., Wu, Z., Zhuang, Y., Lin, Z., Li, Z., Li, D., Xing, E. P., Zhang, H., Gonzalez, J. E. and Stoica, I. (2023). Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena. NeurIPS 2023. Read source