Keep an LLM evaluation suite useful after the first model choice

An evaluation suite starts losing relevance when the product changes around it. A policy is updated, a new tool is added, requests become longer or a judge begins interpreting the rubric differently. A stable model score can hide those changes.

Maintenance means preserving comparable evidence while checking whether the work you measure still resembles the work users need. The operating process below is an editorial recommendation, using a hypothetical support assistant. It does not claim a particular release cadence or threshold is right for every team.

Give the evidence an owner

Assign someone responsibility for the task contract, dataset, scorer and release report. They need authority to investigate broken cases and stop a misleading comparison from becoming a release claim. Application engineers, domain reviewers and operations staff may contribute without leaving ownership ambiguous.

Define review triggers: a model change, a prompt edit, a policy revision, a retrieval update, a tool-schema change or an incident. Add a periodic review appropriate to the rate and consequence of change. “Continuous” should mean evidence remains part of engineering work, not that every test runs on every keystroke.

OpenAI's evaluation guidance recommends evaluating continuously, monitoring for new cases and expanding the evaluation set over time. Treat that as a process principle; choose your own run schedule and acceptance rules. Evaluation best practices.

Record who can approve a release, who adjudicates disputed labels and who owns remediation. Without these roles, an evaluation can identify a regression that nobody is responsible for resolving.

Version every moving part

A score should identify the candidate configuration, prompt, dataset, reference labels, context builder, tool definitions and scorer. For an agent, add the environment and policy versions. Store raw outputs so you can distinguish a behavioural change from a grading change.

Keep released artefacts immutable. If a label is corrected, create a new version with the case ID, evidence, reason and affected runs. Re-score comparable stored outputs where possible. Do not silently replace a reference and leave old charts looking directly comparable.

For a hypothetical example, policy version two adds a return condition absent from version one. A response can be correct under the old policy and incomplete under the new one. The regression report needs to say which task changed, rather than blaming the model for a contradiction in the evaluation.

Preserve historical task versions when they matter for audit, but mark obsolete cases clearly. A large suite containing contradictory policy eras without labels is harder to trust than a smaller, documented suite.

Use layers with different jobs

Maintain a small diagnostic set for quick iteration, a stable regression suite for known requirements, a representative sample for current workload evidence and challenge sets for consequential edge cases. Protect a final holdout for important selection decisions.

These layers need not run at the same frequency. A prompt edit might trigger fast objective checks and selected regression cases. A model migration may need the full workload comparison, judge audit, latency measurement and agent recovery suite.

Keep the layers separate in reporting. A public regression set that developers know well remains valuable for catching reintroduced failures. It is weaker evidence of fresh generalisation. Use the holdout controls when making a new selection claim.

Choose execution breadth from what changed. If a context builder alters every request, a narrow format-only check is insufficient. If a documentation label changes, rerunning a large inference experiment may add no useful evidence. Record the rationale for the checks selected.

Triage a regression before changing the model

Start by identifying whether the input, output or score changed. Compare stored requests and responses under the same scorer. Check dataset and policy versions, tool availability, parser behaviour and execution errors.

If the same output receives different grades, investigate the scorer or reference before claiming candidate degradation. If the input changed, ask whether the new test represents a deliberate product requirement. If the environment failed, distinguish infrastructure reliability from answer quality while retaining both in the operating report.

For a real behavioural regression, inspect case-level differences and important slices. Determine whether the failure violates a gate or represents a tradeoff in a preference measure. Use the uncertainty procedure for close comparisons rather than treating every small fluctuation as a release blocker.

Assign a disposition: fix, narrow scope, accept with an explicit reason or gather more evidence. Preserve the failed result and the reason for the decision. Repeatedly rerunning until a configuration passes is not a resolution.

Turn incidents into useful cases

Capture an incident's request, available evidence, expected behaviour and observed failure. Minimise sensitive content and retain enough structure to reproduce the mechanism. Have a domain reviewer validate the expected outcome.

For the hypothetical assistant, an incident might reveal that an order record and a policy document use different date conventions. Add the specific case to the regression suite and create reviewed variants to test the broader mechanism. Keep those related cases grouped.

Do not let incident collection replace representative sampling. Incidents are deliberately failure-heavy and often depend on what users notice or report. They are excellent diagnostic evidence but do not, by themselves, estimate the frequency of all failures.

Track which change fixed the mechanism and which checks demonstrate it. For state-changing systems, include the agent trajectory and recovery checks, not merely a corrected final message.

Reassess the grader and workload

Audit automated judges after rubric, model or task changes and on a recurring sample of fresh cases. Check both flagged failures and ordinary passes against human labels. The human and model grading guide explains why reviewing only disagreements can miss false passes.

Compare recent traffic with the dataset's composition: task families, context length, languages, missing-information rates and tool paths. Use these observations to decide what to sample next. A changing input mix is a reason to investigate, not automatic proof that quality has fallen.

Report current representative results alongside stable-core results. The former addresses today's workload; the latter helps attribute changes on a common task set. Explain new weights or sampling rules instead of splicing unlike scores into one trend line.

Write a decision record with an expiry trigger

Each consequential release record should name the objective, versions, results, critical failures, uncertainty, operating cost, unresolved limits and approval owner. State what action follows and what evidence would cause the team to revisit it.

For example, a hypothetical decision could permit operator-reviewed policy answers while excluding autonomous refunds, pending recovery evidence. That is a bounded operating choice, not a declaration that the model is generally safe or best.

Keep earlier records and the original outputs available for later review. An evaluation suite earns its place by improving decisions and explaining failures over time. Return to the benchmarking guide whenever the product's decision contract changes, so maintenance continues to measure the job the system is actually expected to do.