Sol versus Luna: what happened when two AI agents built an editorial cluster
On this page
GPT-6.1 Sol produced the preferred article cluster in Digital Sanctum’s first editorial benchmarking pilot. GPT-6 Luna finished substantially sooner. Those findings describe different parts of the workflow, and neither alone establishes overall productivity or general model superiority.
Both sessions received the same commission: research and write a practical cluster about AI model benchmarking, with one pillar, ten supporting articles and a complete editorial handoff. Both used high reasoning effort and goal execution. The methodology explains the controls and the frozen scoring rules.
The quality decision came from one blinded GPT-6.1 Sol judge, the same exact model designation as the winning contestant. There was no independent human validation and only one production run per model. Treat the scores as evidence-backed editorial judgements about these submissions, with that limitation in view.
The result at a glance
| Recorded outcome | GPT-6.1 Sol | GPT-6 Luna |
|---|---|---|
| Frozen quality score | 92.05 / 100 | 85.93 / 100 |
| Complete articles | 11 | 11 |
| Elapsed production time, rounded | 18 min 18 sec | 9 min 28 sec |
| Input tokens | 3,046,820 | 3,110,420 |
| Cached input, included above | 2,894,720 | 3,009,152 |
| Output tokens | 41,141 | 47,354 |
| Reasoning output, included above | 5,500 | 9,353 |
| Total tokens, input plus output | 3,087,961 | 3,157,774 |
| Billed monetary cost | Not measured | Not measured |
The unrounded elapsed times were 1,097.906772 seconds and 567.973218 seconds. Luna took 48.27% less production time in this run and used 2.26% more total tokens. More tokens do not establish a higher bill: cached usage, charging rules and unmeasured fees prevent that inference. The usage record preserves the categories and measurement notes.
Sol’s displayed lead is 6.12 score points. The underlying scores were 92.0454545 and 85.9272727 before rounding. That gap is a rubric-based judgement, not a measured probability, confidence interval or claim of statistical significance.
What both sessions delivered
Each agent delivered one pillar, ten supporting drafts, eleven briefs, an editorial strategy, cluster map, source register, link map, publication sequence and handoff. Structural checks found the required draft count, usable metadata, resolved original local links and lengths within the shared requirements under the common counting approach.
The clusters addressed the requested coverage in their own editorial structures. Both discuss public benchmark interpretation, task design, leakage, scoring, human and model judges, uncertainty, fair comparisons, cost, latency and agent evaluation. Neither was scored merely for being longer or finishing first.
The frozen review did not establish a fabricated citation, missing full draft or other blocker under the predefined blocker categories. That finding did not mean every source summary was accurate or every draft was ready to publish. Required corrections remained, particularly for Luna.
Why the judge preferred Sol
Sol’s stronger work linked the series through a recurring support-assistant example and gave readers more complete analytical methods. Its drafts included usable classification arithmetic, paired uncertainty reasoning, task-level cost calculations and a concrete recovery scenario for an ambiguous state-changing tool call.
That combination helped the cluster move from general advice to decisions a practitioner could follow. The judge preferred Sol in both presentation orders, with ties in writing quality and completeness. These were two passes by the same reviewer using the same evidence, not independent replications.
The largest differences concerned source precision, practical depth and coherence across the collection. Readers can inspect the article scores, cluster scores and full review package, rather than relying on an unexplained overall ranking.
What required correction in Luna’s publication copies
Two drafts attributed the position-bias paper linked at arXiv 2406.07791 to Zheng and colleagues. The linked paper is by Lin Shi and colleagues. Publication copies now name Shi et al. and pin the revised 2025 version, while noting its original 2024 submission. The paper itself was real; the author label was wrong. See the primary paper record.
Two other passages described the rephrased-samples contamination paper as an original-versus-rewritten inference probe. Its central method instead examines training on rephrased or translated benchmark samples, their ability to evade overlap detection and a decontamination approach. We corrected that summary to match the primary study. This was a source-fidelity correction, not evidence that the citation was fabricated.
The publication pass also made groundedness rubric levels mutually exclusive, removed ambiguous retry-cost accounting and completed the hypothetical cost-per-success worksheet. The uncertainty article gained fully specified hypothetical paired outcomes and executable resampling code. Judge routing now refers to evidence gaps, disagreement and risk, with confidence signals requiring validation. An unsupported fatigue assertion was reframed as an editorial process recommendation.
These changes improve the public articles. They do not revise the frozen scores. The original submission remains available so a reader can distinguish the generated work from its later publication copy.
What the timing does and does not tell us
Luna’s shorter production time is a measured result of this run. Its complete draft set was ready in about nine and a half minutes, compared with Sol’s eighteen and a third. That is useful when considering turnaround for a similar commission.
The reviewer estimated 3–5 person-hours of essential editorial work for Sol and 7–12 for Luna, plus optional improvements. These ranges were estimates made during the review. Editing work was not timed as an experiment, so they are not measured labour savings. Website integration, builds and publication operations were excluded from those estimates.
We therefore cannot conclude that either model had a lower end-to-end cost per published cluster. Adding estimated editing hours to measured production seconds would still leave a forecast, not an observed productivity result. Reviewer usage—8,402,872 reported tokens—is separate evaluation overhead and is not attributed to either producer.
Limits readers should keep in view
This pilot has one generated cluster per candidate and one judge. The eleven articles within each cluster share a production session and editorial plan; they are not independent model samples. There is no basis here for a significance test or a confidence interval about general model performance.
The reviewer received anonymous A/B packages before identities and efficiency were revealed. That reduces direct identity cues, but style can still reveal clues. More seriously, the judge shared the exact model designation of Sol. Blinding cannot remove shared model preferences or substitute for human editorial judgement.
The 71-entry fact-check ledger records targeted primary-source checks. It does not certify every sentence, reproduce the research or establish that changing documentation will remain accurate. Both the production and review records are dated 22 October 2026. Repeating this task later may produce different work.
Read both clusters and inspect the evidence
Sol’s cluster is the featured blog series, starting with AI model benchmarking: a practical guide. Luna’s complete publication collection sits in the runner-up comparison archive. Both identify their generating model and link to the scored originals.
The experiment hub brings together the methodology, scores, usage records, original submissions and publication edit log. Our next stronger experiment would add independent runs, multiple topics, qualified human reviewers and observed editing time. For this pilot, the justified conclusion is narrower: Sol’s submitted cluster was preferred under the frozen rubric, while Luna finished sooner.