Beyond the leaderboard: how we benchmark AI models for editorial work

What happens when two AI agents receive the same editorial commission and have to deliver a complete collection of useful articles? We tested that question with GPT-6.1 Sol and GPT-6 Luna on 22 October 2026. Each was asked to act as an editorial team and create an article cluster about AI model benchmarking.

The experiment compares the submissions those two sessions produced. It does not establish which model is best for every writing task. This article records the method, including the limitations, so readers can inspect the result and adapt the approach to their own work.

The production brief and scoring rubric were frozen before production. This explanatory article was prepared after the experiment; it was not a public preregistration. The experiment hub contains the brief, rubric, scored originals, review and usage records.

Start with a real editorial commission

Both agents received the same audience: technical founders, engineering leads and practitioners choosing or evaluating AI models. Each had to research, plan, brief, draft, fact-check and perform one final editorial pass.

The required output was one pillar article of 1,800–2,200 body words and ten supporting articles of 1,000–1,500 body words each. Outlines alone did not qualify. Deliverables also included an editorial strategy, cluster map, eleven article briefs, source register, internal-link map, publication sequence and handoff.

The brief prescribed coverage, including evaluation datasets, leakage, metrics, human and model judges, uncertainty, fair configurations, cost, latency and agent evaluation. It left the titles, angles, examples and reader journey to each agent. This allowed editorial judgement to influence the result while keeping the commission comparable.

Examples had to be labelled when hypothetical. Sources had to support nearby claims. Australian English, clear explanations and practical usefulness mattered; invented search volumes, customer stories, citations or measurements were prohibited. Read the shared production brief for the complete requirements.

Give both sessions comparable working conditions

Each contestant ran in its own isolated repository checkout from the same starting commit. Both used native Codex goal execution with high reasoning effort, live browsing and the same two starting references. Each could find additional primary sources independently.

Goal mode gave the sessions a concrete completion condition: finish the full editorial handoff and its checks. It was an execution setting, not an evaluation score. Neither session could inspect the other contestant’s work, delegate editorial roles to additional agents or publish its drafts.

We did not set a fixed token or wall-clock budget. No coordinator production follow-up prompts were sent. Neither contestant reported an outage or intervention. These conditions are reported from the execution records and handoffs, rather than claimed as a complete independent audit of every runtime event.

Matching the user brief and selected settings does not equalise internal computation. Models can interpret instructions differently, consume different context and choose different research paths. The result concerns these agent workflows under the recorded conditions, rather than a laboratory isolation of model weights.

Freeze the scoring rules before reading the work

The rubric assigned seven dimensions:

Each dimension used anchored scores from zero to five. Article dimensions were averaged across all eleven articles with equal weight. Cluster dimensions were scored once. Each resulting dimension score was divided by five, multiplied by its weight and summed to give a score out of 100.

The rubric also identified publication blockers: fabricated evidence, materially incorrect guidance and missing full drafts. A strong aggregate could not erase those issues. The frozen rubric preserves the wording, anchors and aggregation rule.

Preserve originals before making publication edits

After both sessions finished, we copied their submissions into frozen directories and recorded SHA-256 hashes. Structural checks covered the expected article count, parseable data, resolved local links and common word-count rules. Both delivered the required eleven complete articles within the requested ranges.

A separate review packet labelled the clusters A and B. Completion timestamps were removed from the review copies and file modification times normalised. The original submission bytes were retained. The reviewer initially received neither the identity mapping nor production time and token figures.

Blinding reduces direct identity and efficiency cues. It cannot remove stylistic clues or guarantee impartial judgement. Preserving the original work also matters: otherwise later editorial repairs could accidentally become the work being scored.

Review quality before revealing speed and usage

A fresh GPT-6.1 Sol session reviewed the anonymous clusters. Memory loading was disabled. It scored articles and clusters against the frozen rubric, maintained a 71-entry targeted fact-check ledger and compared the clusters in both presentation orders. It also proposed corrections and estimated editorial effort.

The coordinator checked the required review files and froze the quality scores and decision before revealing identities and efficiency measurements. The same judge then completed the efficiency analysis without changing the frozen quality scores.

The judge was a single model, with the same exact model designation as the eventual winner. This is a material limitation. A fresh session is separate from the producer, but it is not independent human validation. Reversing presentation order is an internal check by that judge, not a second independent review. Source checks were targeted, not an exhaustive verification of every sentence or a reproduction of cited studies.

Count tokens without adding subsets twice

We collected runtime-reported cumulative counters externally. The final cumulative snapshot for each session was used; cumulative snapshots were not summed. Recorded categories include input, cached input, output, reasoning output and total tokens.

Cached input is included in input. Reasoning output is included in output. Therefore total tokens equal input plus output: neither subset is added a second time. These totals include repeated context and are not counts of unique words or the finished articles alone.

Elapsed production time runs from the production start to the recorded completed turn, including tools and waiting. Token counters and elapsed time describe production only. Reviewer usage is reported separately as overhead. No billed monetary task cost was measured, and we do not turn subscription allowances or token totals into an invented dollar figure.

Publish the outcome with its evidence

The policy was to feature the winning cluster on the Digital Sanctum blog and place the runner-up in a separate comparison archive. Both receive authorship disclosures, source checks and publication edits. Scores remain attached to the frozen submissions; edited publication copies are not silently rescored.

Readers can download both original packages and inspect the publication edit log. It records source corrections, practical clarifications, link changes and publication metadata. Estimated editing hours from the review are estimates, not measured labour or realised savings.

For stronger future evidence, repeat the experiment across several independent topics and runs, use multiple blinded judges including qualified humans, and time the actual editing work. Eleven related articles from one session are not eleven independent model runs. This pilot supports a decision about these two submitted clusters; it supports neither statistical significance nor general model superiority.

Read Sol versus Luna: the results to see what happened.