Sol, Luna and DeepSeek: what our three-model editorial benchmark found
On this page
GPT-6.1 Sol’s submission ranked first, GPT-6 Luna’s second and DeepSeek V4.1 Flash’s third in a fresh blinded review of three complete article clusters about AI model benchmarking. DeepSeek delivered all eleven drafts and a useful editorial handoff, but its original cost article needed material corrections before publication.
This extends our 22 October Sol–Luna experiment. We retained those original drafts, added one DeepSeek submission and commissioned a fresh three-way quality review. The original scores remain 92.05 for Sol and 85.93 for Luna. Different scores in this new round reflect a separate judgement, not a change to those models or their original work.
The three submissions and their scores
Scores apply to frozen eleven-article submissions. One production run per producer and one Sol judge; the judge shares the winner’s model designation. No statistical uncertainty interval or general model ranking is established.
Download: SVG · PNG · CSV · Link to figure · Full-precision chart data
View chart data (display rounded)
| Producer | Whole-cluster score / 100 |
|---|---|
| GPT-6.1 Sol | 94.27 |
| GPT-6 Luna | 80.64 |
| DeepSeek V4.1 Flash | 73.82 |
Sources: Frozen scores · Run identities.
| Producer | New review score / 100 | Recorded production elapsed time | Recorded production tokens |
|---|---|---|---|
| GPT-6.1 Sol | 94.27 | 18m 18s | 3,087,961 |
| GPT-6 Luna | 80.64 | 9m 28s | 3,157,774 |
| DeepSeek V4.1 Flash | 73.82 | 58m 38s | 32,056,365 |
Each candidate produced one pillar and ten supporting articles, plus briefs, source records and an editorial plan. Eleven related drafts are one production run per candidate, rather than eleven independent experiments.
Elapsed time includes tools and waiting. DeepSeek’s run included command-approval waits before we enabled permission bypass at the user’s request; those waits were not timed separately. Its native goal clock was 58m 35s and launch-to-completion time 58m 38s. These observations do not isolate inference speed or establish how quickly another operator would complete the same work.
The same frozen rubric allocated accuracy and sources 25%, cluster strategy 20%, practical usefulness 20%, writing 15%, coherence and links 10%, intent and metadata 5%, and completeness 5%. Four dimensions were scored per article and averaged equally across eleven articles; three were scored once per cluster. The coordinator recalculated the weighted totals from the frozen records.
All dimensions use a shared 0–5 scale; labels include rubric weights. Accuracy, usefulness, writing and intent are means across eleven drafts; strategy, coherence and completeness are scored once per cluster. Missing scores are refused, never drawn as zero.
Download: SVG · PNG · CSV · Link to figure · Full-precision chart data
View chart data (display rounded)
| Dimension (weight; scoring level) | GPT-6.1 Sol / 5 | GPT-6 Luna / 5 | DeepSeek V4.1 Flash / 5 |
|---|---|---|---|
| Accuracy & sources (25%; eleven-draft mean) | 5.00 | 4.00 | 3.36 |
| Cluster strategy (20%; cluster) | 5.00 | 4.00 | 4.00 |
| Practical usefulness (20%; eleven-draft mean) | 4.36 | 3.91 | 3.64 |
| Writing quality (15%; eleven-draft mean) | 4.00 | 4.00 | 3.82 |
| Coherence & links (10%; cluster) | 5.00 | 4.00 | 3.00 |
| Intent & metadata (5%; eleven-draft mean) | 4.82 | 4.00 | 4.00 |
| Completeness (5%; cluster) | 5.00 | 5.00 | 5.00 |
Sources: Frozen scores · Run identities.
| Dimension | Weight | Sol / 5 | Luna / 5 | DeepSeek / 5 |
|---|---|---|---|---|
| Accuracy and sources | 25% | 5.00 | 4.00 | 3.36 |
| Cluster strategy | 20% | 5.00 | 4.00 | 4.00 |
| Practical usefulness | 20% | 4.36 | 3.91 | 3.64 |
| Writing quality | 15% | 4.00 | 4.00 | 3.82 |
| Coherence and links | 10% | 5.00 | 4.00 | 3.00 |
| Intent and metadata | 5% | 4.82 | 4.00 | 4.00 |
| Completeness | 5% | 5.00 | 5.00 | 5.00 |
Displayed dimension scores are rounded; totals use the unrounded values. The article scores and cluster scores retain the detail.
What the reviewer preferred
Sol’s strongest material turned evaluation advice into worked procedures: a confusion matrix, a Wilson interval with paired resampling, and recovery from an ambiguous acknowledgement of a state-changing agent action. It remains the featured AI model benchmarking series.
Luna offered clear explanations and cautious recommendations, but needed more completed examples and citation repairs. Its originals misattributed a real position-bias paper to different authors. The paper is by Lin Shi and colleagues; the reviewer distinguished that error from fabricated evidence. Luna’s edited comparison archive remains available with its original-round score and edit history.
DeepSeek supplied useful schemas, formulas and a broad source catalogue. Its weaker guidance treated nominal effort labels as matched computation, gave categorical sample-size advice and used inconsistent holdout-retirement rules. Its cost draft had two publication holds:
- It defined output as visible text, then used output in a formula that needed total billed output. OpenAI’s reasoning documentation includes reasoning in output charges.
- Its Haiku 5.5 table omitted the prompt-length bracket. Anthropic’s pricing page, checked 26 October 2026, distinguishes prompts through 100K tokens from longer prompts.
The DeepSeek publication archive contains revised copies with an edit log. Those revisions do not change the original score or replace the scored download.
How the fresh review worked
A fresh GPT-6.1 Sol session used Codex CLI 0.163.0, high effort and live browsing to review anonymised packets. It read all 33 articles, recorded 162 substantive claim checks, and made 176 bibliographic checks against 49 live source records. Each article received at least three substantive checks. These were targeted checks, not a reproduction of the cited experiments or a verification of every sentence.
Per-submission scores were frozen before pairwise comparisons. All six comparison directions preserved the overall ranking, with some question-level ties. They are consistency checks within one judge’s review, not independent replications. Two technical coordinator notes addressed complete reading of truncated tool output and UTC versus Melbourne date labels; neither supplied producer identities, efficiency figures or candidate-specific quality feedback.
The complete quality archive was checksum-verified and frozen before the coordinator combined scores with producer identities and usage. The reviewer remains blinded; the merge added no quality scores. The evidence hub links the review, claim ledger, identity mapping and freeze record.
The judge shares Sol’s exact model designation and family. Blinding cannot remove every stylistic clue or model-family preference. This is model-assisted editorial judgement, without independent human validation, repeated production runs or statistical evidence of general superiority.
Recorded cache usage and estimated token costs
| Producer | Input tokens | Cached input, included in input | Output tokens | API-equivalent estimate, USD |
|---|---|---|---|---|
| Sol | 3,046,820 | 2,894,720 | 41,141 | $1.0051 |
| Luna | 3,110,420 | 3,009,152 | 47,354 | $0.0639 |
| DeepSeek | 31,878,376 | 31,187,584 | 177,989 | $0.3040 |
Cache counts come from recorded runtime usage. We did not estimate cache hits from a presumed percentage. Cached input represented approximately 95.0%, 96.7% and 97.8% of the respective input totals. Total usage is input plus output; neither cached input nor a reasoning subtotal is added again.
The dollar column is a dated token-only reference calculation, not an invoice. Using rates checked 26 October 2026, it applies OpenAI Standard pricing to Sol and Luna and DeepSeek’s direct-API off-peak rates to the CommandCode run. Sunday 25 October UTC was outside DeepSeek’s peak window. The latter is a reference estimate for recorded usage, not proof of CommandCode’s commercial charge or a matched provider comparison.
The fresh judge used 14,735,836 tokens separately from production. Its token-only estimate is $3.9671, applying OpenAI’s long-context uplift to 26 of its 65 recorded requests. Pricing depends on each request’s input length, not the accumulated session total. Setup, tools, publication work, taxes, subscription allocations and unavailable background usage are excluded. See the worked cost guide and request-level calculation.
Sol and Luna reported reasoning subtotals of 5,500 and 9,353, already included in output. DeepSeek’s separate reasoning subtotal and unpersisted verifier/background usage were unavailable, rather than zero. Its production totals reconcile 105 unique persisted assistant requests. Different provider tokenizers and reporting conventions limit direct comparisons.
The harness and model-version boundaries
DeepSeek ran through CommandCode CLI 1.88.0, its provider, browsing tools, system prompt, context management and goal supervisor. The earlier contestants used Codex CLI 0.163.0. DeepSeek research occurred on 25 October UTC, while the originals were produced on 22 October. The shared brief, rubric and repository baseline were retained, with additional date and anti-contamination instructions disclosed in the extension record.
These changes mean this is a comparison of recorded agent workflows. We cannot attribute every difference to model weights. “High” effort is a provider-specific setting label, not proof of equal internal computation. CommandCode’s goal verifier was unavailable and accepted the producer’s completion claim; external structural checks and the later blind quality review supplied separate evidence.
The configured model was deepseek/deepseek-v4.1-flash using its latest alias. We have no provider receipt pinning the served checkpoint. DeepSeek’s September release and 17 October 1017 update are separate Timeline events. Their dates, or the current behaviour of DeepSeek’s direct API, do not prove which snapshot CommandCode served for this run.
What this changes for an editorial team
Sol remains the featured cluster under our publication policy. Luna and DeepSeek remain inspectable in comparison archives, with scored originals separated from revised copies. The most useful outcome is the record of where editorial work was needed: billing semantics, source conditions, practical examples and consistency can matter more than a polished opening.
The reviewer estimated 4–7 editor hours for Sol, 8–13 for Luna and 14–24 for DeepSeek. Those are planning estimates, not timed editing work or measured savings. A stronger next experiment would repeat independent topics and runs, add qualified human reviewers and measure actual repair time.
Start with the benchmarking pillar to design your own evaluation, or inspect this round’s evidence to assess the judgement yourself.