Beyond the leaderboard: how we benchmark AI models for editorial work
On this page
What happens when two AI agents receive the same editorial commission and have to deliver a complete collection of useful articles? We tested that question with GPT-6.1 Sol and GPT-6 Luna on 22 October 2026. Each was asked to act as an editorial team and create an article cluster about AI model benchmarking.
The experiment compares the submissions those two sessions produced. It does not establish which model is best for every writing task. This article records the method, including the limitations, so readers can inspect the result and adapt the approach to their own work.
The production brief and scoring rubric were frozen before production. This explanatory article was prepared after the experiment; it was not a public preregistration. The experiment hub contains the brief, rubric, scored originals, review and usage records.
Start with a real editorial commission
Both agents received the same audience: technical founders, engineering leads and practitioners choosing or evaluating AI models. Each had to research, plan, brief, draft, fact-check and perform one final editorial pass.
The required output was one pillar article of 1,800–2,200 body words and ten supporting articles of 1,000–1,500 body words each. Outlines alone did not qualify. Deliverables also included an editorial strategy, cluster map, eleven article briefs, source register, internal-link map, publication sequence and handoff.
The brief prescribed coverage, including evaluation datasets, leakage, metrics, human and model judges, uncertainty, fair configurations, cost, latency and agent evaluation. It left the titles, angles, examples and reader journey to each agent. This allowed editorial judgement to influence the result while keeping the commission comparable.
Examples had to be labelled when hypothetical. Sources had to support nearby claims. Australian English, clear explanations and practical usefulness mattered; invented search volumes, customer stories, citations or measurements were prohibited. Read the shared production brief for the complete requirements.
Give both sessions comparable working conditions
Each contestant ran in its own isolated repository checkout from the same starting commit. Both used native Codex goal execution with high reasoning effort, live browsing and the same two starting references. Each could find additional primary sources independently.
Goal mode gave the sessions a concrete completion condition: finish the full editorial handoff and its checks. It was an execution setting, not an evaluation score. Neither session could inspect the other contestant’s work, delegate editorial roles to additional agents or publish its drafts.
We did not set a fixed token or wall-clock budget. No coordinator production follow-up prompts were sent. Neither contestant reported an outage or intervention. These conditions are reported from the execution records and handoffs, rather than claimed as a complete independent audit of every runtime event.
Matching the user brief and selected settings does not equalise internal computation. Models can interpret instructions differently, consume different context and choose different research paths. The result concerns these agent workflows under the recorded conditions, rather than a laboratory isolation of model weights.
Freeze the scoring rules before reading the work
The rubric assigned seven dimensions:
| Dimension | Weight | Scoring unit |
|---|---|---|
| Accuracy and sources | 25% | Each article |
| Cluster strategy | 20% | Whole cluster |
| Practical usefulness | 20% | Each article |
| Writing quality | 15% | Each article |
| Coherence and links | 10% | Whole cluster |
| Intent and metadata | 5% | Each article |
| Completeness | 5% | Whole cluster |
Each dimension used anchored scores from zero to five. Article dimensions were averaged across all eleven articles with equal weight. Cluster dimensions were scored once. Each resulting dimension score was divided by five, multiplied by its weight and summed to give a score out of 100.
The rubric also identified publication blockers: fabricated evidence, materially incorrect guidance and missing full drafts. A strong aggregate could not erase those issues. The frozen rubric preserves the wording, anchors and aggregation rule.
Preserve originals before making publication edits
After both sessions finished, we copied their submissions into frozen directories and recorded SHA-256 hashes. Structural checks covered the expected article count, parseable data, resolved local links and common word-count rules. Both delivered the required eleven complete articles within the requested ranges.
A separate review packet labelled the clusters A and B. Completion timestamps were removed from the review copies and file modification times normalised. The original submission bytes were retained. The reviewer initially received neither the identity mapping nor production time and token figures.
Blinding reduces direct identity and efficiency cues. It cannot remove stylistic clues or guarantee impartial judgement. Preserving the original work also matters: otherwise later editorial repairs could accidentally become the work being scored.
Review quality before revealing speed and usage
A fresh GPT-6.1 Sol session reviewed the anonymous clusters. Memory loading was disabled. It scored articles and clusters against the frozen rubric, maintained a 71-entry targeted fact-check ledger and compared the clusters in both presentation orders. It also proposed corrections and estimated editorial effort.
| Stage | Original two-model round | Later three-model extension |
|---|---|---|
| Shared commission | Shared brief and frozen rubric | Same brief and rubric; original Sol/Luna submissions retained |
| Production | Independent Sol and Luna sessions, 22 October | DeepSeek extension, 25 October; different provider and CLI |
| Evidence preservation | Original files preserved and checksummed | New submission preserved; originals not regenerated |
| Anonymous quality review | A = Sol; B = Luna, revealed after quality freeze | A = Luna; B = Sol; C = DeepSeek, revealed after quality freeze |
| Quality freeze | Original two-model scores frozen | Fresh three-way scores frozen; not a revision of the original scores |
| Identity and efficiency | Reveal producer identities; attach recorded production usage | Reveal new round’s identities; attach recorded production usage |
| Publication copies | Disclosed editorial corrections; original downloads retained | Disclosed extension and later figures; scored submissions stay unchanged |
Both rounds used a single GPT-6.1 Sol model judge, sharing the winning producer’s exact model designation. A separate session and order checks are not independent human validation. This diagram records history; it does not claim proposed future controls were performed.
The recorded editorial review sequence, with original and extension rounds kept distinct.
Source: this article, “Review quality before revealing speed and usage”; illustration definitions and source hashes. Later editorial addition: separate edit record; frozen submissions and scores are unchanged.
The coordinator checked the required review files and froze the quality scores and decision before revealing identities and efficiency measurements. The same judge then completed the efficiency analysis without changing the frozen quality scores.
The judge was a single model, with the same exact model designation as the eventual winner. This is a material limitation. A fresh session is separate from the producer, but it is not independent human validation. Reversing presentation order is an internal check by that judge, not a second independent review. Source checks were targeted, not an exhaustive verification of every sentence or a reproduction of cited studies.
Count tokens without adding subsets twice
We collected runtime-reported cumulative counters externally. The final cumulative snapshot for each session was used; cumulative snapshots were not summed. Recorded categories include input, cached input, output, reasoning output and total tokens.
Cached input is included in input. Reasoning output is included in output. Therefore total tokens equal input plus output: neither subset is added a second time. These totals include repeated context and are not counts of unique words or the finished articles alone.
Elapsed production time runs from the production start to the recorded completed turn, including tools and waiting. Token counters and elapsed time describe production only. Reviewer usage is reported separately as overhead. No billed monetary task cost was measured, and we do not turn subscription allowances or token totals into an invented dollar figure.
Publish the outcome with its evidence
The policy was to feature the winning cluster on the Digital Sanctum blog and place the runner-up in a separate comparison archive. Both receive authorship disclosures, source checks and publication edits. Scores remain attached to the frozen submissions; edited publication copies are not silently rescored.
Readers can download both original packages and inspect the publication edit log. It records source corrections, practical clarifications, link changes and publication metadata. Estimated editing hours from the review are estimates, not measured labour or realised savings.
For stronger future evidence, repeat the experiment across several independent topics and runs, use multiple blinded judges including qualified humans, and time the actual editing work. Eleven related articles from one session are not eleven independent model runs. This pilot supports a decision about these two submitted clusters; it supports neither statistical significance nor general model superiority.
Read Sol versus Luna: the results to see what happened.
Method extension · 26 October 2026
The sections above describe the original 22 October two-model round. The extension added one DeepSeek V4.1 Flash submission, produced on 25 October UTC / 26 October Melbourne, and a fresh three-way review of all frozen originals. Historical 92.05/85.93 scores remain unchanged; the new review gives Sol 94.27, Luna 80.64 and DeepSeek 73.82.
The DeepSeek agent used CommandCode CLI 1.88.0 and its configured latest alias, while the earlier producers used Codex CLI 0.163.0. The shared production brief, rubric and repository baseline were retained, with additional date and anti-contamination instructions. We did not rerun the original producers. Provider, tools, system prompt, context management, research date and supervisor therefore changed together. Nominal high-effort labels do not match internal computation. No provider receipt proves a 1017 served snapshot.
During production the user requested permission bypass because approval prompts interrupted work. The change and earlier waits are disclosed; the waits were not separately timed. CommandCode’s goal verifier was unavailable and accepted the producer’s completion claim. External structure checks and the later quality review supplied separate evidence of completeness and quality.
A fresh GPT-6.1 Sol judge read all 33 articles, made 162 substantive checks and 176 bibliographic checks against 49 live sources. Per-submission scores were frozen before all six pairwise directions. Two technical coordinator interventions addressed complete reading of truncated tool output and UTC/Melbourne date labels without identity or efficiency disclosure. The quality archive was externally frozen before the coordinator combined identities and usage; the judge remains blinded. This procedure limits direct cues but cannot remove model-family preferences or substitute for qualified human review.
Frozen producer counters remain the production boundary: later follow-ups in the same native sessions are excluded. Actual cached-input counters are used, rather than estimated cache rates. DeepSeek’s 105 persisted requests reconcile to its totals; separate reasoning and unpersisted verifier/background usage remain unavailable. The fresh judge’s usage is separate overhead. New dated API-equivalent dollar calculations are estimates, with per-request long-context rules and explicit exclusions; the original round’s statement that no billed task cost was measured remains true.
The featured Sol series and Luna archive retain their history. Revised DeepSeek copies are published in a separate comparison archive with a source-to-publication edit log. No publication copy is silently rescored. Read the new findings and inspect the extension evidence.
Separate four-way round · 28 October 2026
The four-model follow-up adds one independently produced Terra cluster to the three preserved original submissions. A fresh Sol high judge on Codex CLI 0.163.0 read 44 drafts and 74 supporting records, recorded 193 substantive checks plus two bibliographic claim records and 127 citation associations, and froze scores before twelve same-judge direction checks. The coordinator independently verified 118 candidate hashes, completion/freeze records, recorded read ranges and weighted arithmetic, then joined identities and recorded efficiency only after an external quality freeze. These documentary checks are not exhaustive factual verification, independent human acceptance or proof that blinding removed every clue.
New scores are Sol 93.81818181818181, Terra 81.0, Luna 80.63636363636364 and DeepSeek 78.18181818181819. Terra–Luna is an editorial near-tie. The previous two-way and three-way rounds remain unchanged, including their assets and downloads. No new production runs or rescoring of publication copies took place. Inspect the four-way evidence and edited Terra archive. Different dates, tools, CLI/context handling, supervision and tokenizers limit causal comparison; nominal effort is not matched compute.
Five-model follow-up · 29 October 2026
Grok 4.7 produced one new eleven-article submission through Grok CLI. A fresh Sol judge reviewed all five preserved submissions with integer anchors before completing same-judge, reversed-order comparisons. Different providers, dates, tools and supervision remain confounders. The reviewer had finished before the coordinator inadvertently displayed part of the identity mapping before external comparison freeze; frozen candidates and scores were verified unchanged. The review is advisory and human technical and editorial acceptance remains outstanding. Read the five-way results.