{
  "checks": [
    {
      "check_id": "A-P01-01",
      "cluster": "A",
      "article_id": "P01",
      "article_location": "A/articles/ai-model-benchmarking-guide.md:29 (Use public benchmarks to narrow the search)",
      "claim": "MMLU covers 57 subjects, including mathematics, history and law.",
      "source_url": "https://arxiv.org/abs/2009.03300",
      "source_title": "Measuring Massive Multitask Language Understanding",
      "verified_evidence": "Abstract names 57 tasks and those subject examples.",
      "verdict": "supported",
      "severity": "none",
      "accessed_on": "2026-10-22",
      "verification_method": "Primary page/paper inspected",
      "citation_status": "cited_in_article"
    },
    {
      "check_id": "A-P01-02",
      "cluster": "A",
      "article_id": "P01",
      "article_location": "A/articles/ai-model-benchmarking-guide.md:67 (Score what success means)",
      "claim": "OpenAI recommends task-specific evaluation and human calibration of automated scoring.",
      "source_url": "https://developers.openai.com/api/docs/guides/evaluation-best-practices",
      "source_title": "Evaluation best practices | OpenAI API",
      "verified_evidence": "Best-practice bullets recommend workload-specific tests and human-feedback calibration.",
      "verdict": "supported",
      "severity": "none",
      "accessed_on": "2026-10-22",
      "verification_method": "Primary page/paper inspected",
      "citation_status": "cited_in_article"
    },
    {
      "check_id": "A-P01-03",
      "cluster": "A",
      "article_id": "P01",
      "article_location": "A/articles/ai-model-benchmarking-guide.md:99 (Measure the cost of completed work)",
      "claim": "$12 across 100 attempts with 80 accepted outcomes gives $0.15 per acceptance.",
      "source_url": "",
      "source_title": "Reviewer arithmetic on labelled hypothetical inputs",
      "verified_evidence": "12 / 80 = 0.15; all attempted work remains in spend. Explicitly hypothetical.",
      "verdict": "supported",
      "severity": "none",
      "accessed_on": "2026-10-22",
      "verification_method": "Independent arithmetic; no empirical result claimed"
    },
    {
      "check_id": "A-S01-01",
      "cluster": "A",
      "article_id": "S01",
      "article_location": "A/articles/read-ai-benchmark-leaderboards.md:19 (Identify the task behind the name)",
      "claim": "HumanEval tests functional correctness of programs generated from docstrings.",
      "source_url": "https://arxiv.org/abs/2107.03374",
      "source_title": "Evaluating Large Language Models Trained on Code",
      "verified_evidence": "Abstract specifies docstring-to-program synthesis and functional correctness.",
      "verdict": "supported",
      "severity": "none",
      "accessed_on": "2026-10-22",
      "verification_method": "Primary page/paper inspected",
      "citation_status": "cited_in_article"
    },
    {
      "check_id": "A-S01-02",
      "cluster": "A",
      "article_id": "S01",
      "article_location": "A/articles/read-ai-benchmark-leaderboards.md:29 (Inspect what earns the score)",
      "claim": "At-least-one-success among several samples differs from first-attempt success.",
      "source_url": "https://arxiv.org/html/2107.03374v2#S2.SS1",
      "source_title": "Evaluating Large Language Models Trained on Code",
      "verified_evidence": "Section 2.1 defines pass@k using successful samples among k attempts; selection/budget cautions follow logically.",
      "verdict": "supported",
      "severity": "none",
      "accessed_on": "2026-10-22",
      "verification_method": "Primary page/paper inspected",
      "citation_status": "cited_in_article"
    },
    {
      "check_id": "A-S01-03",
      "cluster": "A",
      "article_id": "S01",
      "article_location": "A/articles/read-ai-benchmark-leaderboards.md:39 (Check whether the rows are comparable)",
      "claim": "Default Verified Bash Only view uses a common mini-SWE-agent environment.",
      "source_url": "https://www.swebench.com/",
      "source_title": "SWE-bench Leaderboards",
      "verified_evidence": "Maintainer page, Bash Only description, expressly identifies default view and shared environment.",
      "verdict": "supported",
      "severity": "none",
      "accessed_on": "2026-10-22",
      "verification_method": "Primary page/paper inspected",
      "citation_status": "cited_in_article"
    },
    {
      "check_id": "A-S01-04",
      "cluster": "A",
      "article_id": "S01",
      "article_location": "A/articles/read-ai-benchmark-leaderboards.md:49 (Examine coverage and aggregation)",
      "claim": "HELM evaluates scenarios across dimensions including accuracy, robustness and efficiency.",
      "source_url": "https://arxiv.org/abs/2211.09110",
      "source_title": "Holistic Evaluation of Language Models",
      "verified_evidence": "Abstract lists a multi-metric scenario framework and these dimensions.",
      "verdict": "supported",
      "severity": "none",
      "accessed_on": "2026-10-22",
      "verification_method": "Primary page/paper inspected",
      "citation_status": "cited_in_article"
    },
    {
      "check_id": "A-S02-01",
      "cluster": "A",
      "article_id": "S02",
      "article_location": "A/articles/design-llm-evaluation-tasks.md:49 (Build a coverage matrix)",
      "claim": "CheckList uses capabilities and test types to expose behaviour beyond held-out accuracy.",
      "source_url": "https://aclanthology.org/2020.acl-main.442/",
      "source_title": "Beyond Accuracy: Behavioral Testing of NLP Models with CheckList",
      "verified_evidence": "ACL paper abstract describes the capability-by-test-type matrix.",
      "verdict": "supported",
      "severity": "none",
      "accessed_on": "2026-10-22",
      "verification_method": "Primary page/paper inspected",
      "citation_status": "cited_in_article"
    },
    {
      "check_id": "A-S02-02",
      "cluster": "A",
      "article_id": "S02",
      "article_location": "A/articles/design-llm-evaluation-tasks.md:59 (Include negative and ambiguous cases)",
      "claim": "Related scenario variants crossing development/test boundaries can undermine generalisation evidence.",
      "source_url": "https://scikit-learn.org/stable/modules/cross_validation.html",
      "source_title": "Cross-validation: evaluating estimator performance",
      "verified_evidence": "Grouped-data section requires separation of dependent groups; scenario grouping is a reasonable editorial application.",
      "verdict": "supported",
      "severity": "none",
      "accessed_on": "2026-10-22",
      "verification_method": "Primary page/paper inspected",
      "citation_status": "independent corroborating source; not a nearby article citation"
    },
    {
      "check_id": "A-S03-01",
      "cluster": "A",
      "article_id": "S03",
      "article_location": "A/articles/evaluation-dataset-leakage-holdouts.md:17 (Distinguish three routes to misleading evidence)",
      "claim": "Leakage uses information unavailable at prediction time; split before fitting transformations.",
      "source_url": "https://scikit-learn.org/stable/common_pitfalls.html",
      "source_title": "Common pitfalls and recommended practices",
      "verified_evidence": "Data-leakage section defines the timing problem and gives split-before-fit advice.",
      "verdict": "supported",
      "severity": "none",
      "accessed_on": "2026-10-22",
      "verification_method": "Primary page/paper inspected",
      "citation_status": "cited_in_article"
    },
    {
      "check_id": "A-S03-02",
      "cluster": "A",
      "article_id": "S03",
      "article_location": "A/articles/evaluation-dataset-leakage-holdouts.md:17 (Distinguish three routes to misleading evidence)",
      "claim": "Training-data detection studies black-box access without access to the pretraining corpus.",
      "source_url": "https://arxiv.org/abs/2310.16789",
      "source_title": "Detecting Pretraining Data from Large Language Models",
      "verified_evidence": "Abstract states that problem and its limited-access setting; the article makes no universal clean-data guarantee.",
      "verdict": "supported",
      "severity": "none",
      "accessed_on": "2026-10-22",
      "verification_method": "Primary page/paper inspected",
      "citation_status": "cited_in_article"
    },
    {
      "check_id": "A-S03-03",
      "cluster": "A",
      "article_id": "S03",
      "article_location": "A/articles/evaluation-dataset-leakage-holdouts.md:27 (Document provenance before splitting)",
      "claim": "Datasheets proposes documenting motivation, composition, collection and uses.",
      "source_url": "https://arxiv.org/abs/1803.09010",
      "source_title": "Datasheets for Datasets",
      "verified_evidence": "Abstract names these documentation categories.",
      "verdict": "supported",
      "severity": "none",
      "accessed_on": "2026-10-22",
      "verification_method": "Primary page/paper inspected",
      "citation_status": "cited_in_article"
    },
    {
      "check_id": "A-S03-04",
      "cluster": "A",
      "article_id": "S03",
      "article_location": "A/articles/evaluation-dataset-leakage-holdouts.md:57 (Audit overlap at several levels)",
      "claim": "Oren et al. compare canonical and shuffled dataset likelihood under exchangeability.",
      "source_url": "https://arxiv.org/abs/2310.17623",
      "source_title": "Proving Test Set Contamination in Black Box Language Models",
      "verified_evidence": "Abstract describes this likelihood test and the exchangeability premise.",
      "verdict": "supported",
      "severity": "none",
      "accessed_on": "2026-10-22",
      "verification_method": "Primary page/paper inspected",
      "citation_status": "cited_in_article"
    },
    {
      "check_id": "A-S04-01",
      "cluster": "A",
      "article_id": "S04",
      "article_location": "A/articles/llm-evaluation-metrics-rubrics.md:37 (Work through a classification example)",
      "claim": "TP=15, FP=10, FN=5, TN=70 yields precision 60%, recall 75%, F1 66.7%, accuracy 85%.",
      "source_url": "https://scikit-learn.org/stable/modules/generated/sklearn.metrics.precision_recall_fscore_support.html",
      "source_title": "precision_recall_fscore_support",
      "verified_evidence": "Definitions checked; reviewer recalculation: 15/25=.6, 15/20=.75, 30/45=.6667, 85/100=.85.",
      "verdict": "supported",
      "severity": "none",
      "accessed_on": "2026-10-22",
      "verification_method": "Primary definitions plus independent arithmetic",
      "citation_status": "cited_in_article"
    },
    {
      "check_id": "A-S04-02",
      "cluster": "A",
      "article_id": "S04",
      "article_location": "A/articles/llm-evaluation-metrics-rubrics.md:37 (Work through a classification example)",
      "claim": "Always predicting no escalation gets 80% accuracy while missing all 20 escalation cases.",
      "source_url": "",
      "source_title": "Reviewer arithmetic on labelled hypothetical inputs",
      "verified_evidence": "80 negatives / 100 cases = 80%; true positives zero. Invented counts, correctly labelled.",
      "verdict": "supported",
      "severity": "none",
      "accessed_on": "2026-10-22",
      "verification_method": "Independent arithmetic; no empirical result claimed"
    },
    {
      "check_id": "A-S04-03",
      "cluster": "A",
      "article_id": "S04",
      "article_location": "A/articles/llm-evaluation-metrics-rubrics.md:37 (Work through a classification example)",
      "claim": "Macro averages weight classes equally; support-weighted summaries differ.",
      "source_url": "https://scikit-learn.org/stable/modules/generated/sklearn.metrics.precision_recall_fscore_support.html",
      "source_title": "precision_recall_fscore_support",
      "verified_evidence": "average parameter defines unweighted label mean and support-weighted mean.",
      "verdict": "supported",
      "severity": "none",
      "accessed_on": "2026-10-22",
      "verification_method": "Primary page/paper inspected",
      "citation_status": "cited_in_article"
    },
    {
      "check_id": "A-S05-01",
      "cluster": "A",
      "article_id": "S05",
      "article_location": "A/articles/human-reviewers-llm-judges.md:47 (Check known bias mechanisms)",
      "claim": "MT-Bench/Chatbot Arena study discusses position, verbosity, self-enhancement and reasoning limits.",
      "source_url": "https://arxiv.org/abs/2306.05685",
      "source_title": "Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena",
      "verified_evidence": "Original abstract explicitly names these limitations; no study agreement percentage is transferred.",
      "verdict": "supported",
      "severity": "none",
      "accessed_on": "2026-10-22",
      "verification_method": "Primary page/paper inspected",
      "citation_status": "cited_in_article"
    },
    {
      "check_id": "A-S05-02",
      "cluster": "A",
      "article_id": "S05",
      "article_location": "A/articles/human-reviewers-llm-judges.md:59 (Measure errors that affect the decision)",
      "claim": "An always-pass judge on 90 passes and 10 failures agrees 90% but detects no failures.",
      "source_url": "",
      "source_title": "Reviewer arithmetic on labelled hypothetical inputs",
      "verified_evidence": "90 correct pass labels / 100 = 90%; none of the 10 failures is detected.",
      "verdict": "supported",
      "severity": "none",
      "accessed_on": "2026-10-22",
      "verification_method": "Independent arithmetic; no empirical result claimed"
    },
    {
      "check_id": "A-S06-01",
      "cluster": "A",
      "article_id": "S06",
      "article_location": "A/articles/benchmark-uncertainty-repeatability.md:27 (Keep the paired outcomes)",
      "claim": "75 both-pass, 10 both-fail, 5 A-only and 10 B-only gives 80 versus 85 successes and 15 discordant cases.",
      "source_url": "",
      "source_title": "Reviewer arithmetic on labelled hypothetical inputs",
      "verified_evidence": "75+10+5+10=100; A=75+5=80; B=75+10=85; discordant=15; difference=5 percentage points.",
      "verdict": "supported",
      "severity": "none",
      "accessed_on": "2026-10-22",
      "verification_method": "Independent arithmetic; no empirical result claimed"
    },
    {
      "check_id": "A-S06-02",
      "cluster": "A",
      "article_id": "S06",
      "article_location": "A/articles/benchmark-uncertainty-repeatability.md:37 (Put a single success rate in context)",
      "claim": "95% Wilson interval for 80/100 is about 71.1%–86.7%.",
      "source_url": "https://www.itl.nist.gov/div898/handbook/prc/section2/prc241.htm",
      "source_title": "7.2.4.1. Confidence intervals",
      "verified_evidence": "Wilson formula inspected; independent substitution yields 0.7111690381 to 0.8666340774. Independence/binomial assumptions are stated.",
      "verdict": "supported",
      "severity": "none",
      "accessed_on": "2026-10-22",
      "verification_method": "Primary formula plus independent arithmetic",
      "citation_status": "cited_in_article"
    },
    {
      "check_id": "A-S06-03",
      "cluster": "A",
      "article_id": "S06",
      "article_location": "A/articles/benchmark-uncertainty-repeatability.md:37 (Put a single success rate in context)",
      "claim": "A frequentist interval concerns long-run procedure coverage rather than future-answer success.",
      "source_url": "https://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.bootstrap.html",
      "source_title": "scipy.stats.bootstrap",
      "verified_evidence": "Documentation explains repeated-sampling coverage, matching the article’s interpretation.",
      "verdict": "supported",
      "severity": "none",
      "accessed_on": "2026-10-22",
      "verification_method": "Primary page/paper inspected",
      "citation_status": "cited_in_article"
    },
    {
      "check_id": "A-S06-04",
      "cluster": "A",
      "article_id": "S06",
      "article_location": "A/articles/benchmark-uncertainty-repeatability.md:47 (Estimate the difference with paired resampling)",
      "claim": "Paired bootstrap uses identical resampled indices for both candidates; percentile interval uses tails.",
      "source_url": "https://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.bootstrap.html",
      "source_title": "scipy.stats.bootstrap",
      "verified_evidence": "paired parameter preserves shared indices; percentile introduction describes the central resampled interval. Group extension and limitations are explicit.",
      "verdict": "supported",
      "severity": "none",
      "accessed_on": "2026-10-22",
      "verification_method": "Primary page/paper inspected",
      "citation_status": "cited_in_article"
    },
    {
      "check_id": "A-S07-01",
      "cluster": "A",
      "article_id": "S07",
      "article_location": "A/articles/fair-ai-model-comparisons.md:27 (Freeze the conditions that define the task)",
      "claim": "Sclar et al. find prompt-format sensitivity in studied few-shot models and weak cross-model format correlation.",
      "source_url": "https://arxiv.org/abs/2310.11324",
      "source_title": "Quantifying Language Models' Sensitivity to Spurious Features in Prompt Design or: How I learned to start worrying about prompt formatting",
      "verified_evidence": "Abstract reports both findings; article limits generalisation to studied settings.",
      "verdict": "supported",
      "severity": "none",
      "accessed_on": "2026-10-22",
      "verification_method": "Primary page/paper inspected",
      "citation_status": "cited_in_article"
    },
    {
      "check_id": "A-S07-02",
      "cluster": "A",
      "article_id": "S07",
      "article_location": "A/articles/fair-ai-model-comparisons.md:47 (Match capabilities without inventing equivalence)",
      "claim": "OpenAI reasoning tokens occupy context and are billed as output despite not being ordinary visible text.",
      "source_url": "https://developers.openai.com/api/docs/guides/reasoning",
      "source_title": "Reasoning models",
      "verified_evidence": "Reasoning documentation states context occupancy and output billing for non-visible reasoning tokens.",
      "verdict": "supported",
      "severity": "none",
      "accessed_on": "2026-10-22",
      "verification_method": "Primary page/paper inspected",
      "citation_status": "cited_in_article"
    },
    {
      "check_id": "A-S07-03",
      "cluster": "A",
      "article_id": "S07",
      "article_location": "A/articles/fair-ai-model-comparisons.md:67 (Save a run manifest someone else can use)",
      "claim": "EleutherAI provides a shared evaluation framework with explicit task/model configuration.",
      "source_url": "https://github.com/EleutherAI/lm-evaluation-harness",
      "source_title": "Language Model Evaluation Harness",
      "verified_evidence": "README basic usage accepts tasks, model arguments, revision and execution settings.",
      "verdict": "supported",
      "severity": "none",
      "accessed_on": "2026-10-22",
      "verification_method": "Primary page/paper inspected",
      "citation_status": "cited_in_article"
    },
    {
      "check_id": "A-S08-01",
      "cluster": "A",
      "article_id": "S08",
      "article_location": "A/articles/ai-latency-cost-per-successful-task.md:17 (Define the task clock)",
      "claim": "Streaming can reduce waiting for visible output.",
      "source_url": "https://developers.openai.com/api/docs/guides/latency-optimization",
      "source_title": "Latency optimization",
      "verified_evidence": "Make your users wait less discusses streaming; article avoids adopting a universal speedup.",
      "verdict": "supported",
      "severity": "none",
      "accessed_on": "2026-10-22",
      "verification_method": "Primary page/paper inspected",
      "citation_status": "cited_in_article"
    },
    {
      "check_id": "A-S08-02",
      "cluster": "A",
      "article_id": "S08",
      "article_location": "A/articles/ai-latency-cost-per-successful-task.md:37 (Account for actual usage)",
      "claim": "Official pricing separates input, output, cached input and tool charges.",
      "source_url": "https://developers.openai.com/api/docs/pricing",
      "source_title": "Pricing | OpenAI API",
      "verified_evidence": "Pricing table headers and Tools section contain separate billing categories.",
      "verdict": "supported",
      "severity": "none",
      "accessed_on": "2026-10-22",
      "verification_method": "Primary page/paper inspected",
      "citation_status": "cited_in_article"
    },
    {
      "check_id": "A-S08-03",
      "cluster": "A",
      "article_id": "S08",
      "article_location": "A/articles/ai-latency-cost-per-successful-task.md:47 (Make cache conditions visible)",
      "claim": "Cache reuse requires matching rendered prefixes and additional eligibility/lookup conditions.",
      "source_url": "https://developers.openai.com/api/docs/guides/prompt-caching",
      "source_title": "Prompt caching",
      "verified_evidence": "Rendered-prefix and lookup sections state these requirements; common text alone need not yield a hit.",
      "verdict": "supported",
      "severity": "none",
      "accessed_on": "2026-10-22",
      "verification_method": "Primary page/paper inspected",
      "citation_status": "cited_in_article"
    },
    {
      "check_id": "A-S08-04",
      "cluster": "A",
      "article_id": "S08",
      "article_location": "A/articles/ai-latency-cost-per-successful-task.md:57 (Calculate cost per accepted task)",
      "claim": "2,000 input at $2/million plus 500 output at $8/million costs $0.008.",
      "source_url": "",
      "source_title": "Reviewer arithmetic on labelled hypothetical inputs",
      "verified_evidence": "2000*2/1e6=.004; 500*8/1e6=.004; sum=.008. Rates expressly invented.",
      "verdict": "supported",
      "severity": "none",
      "accessed_on": "2026-10-22",
      "verification_method": "Independent arithmetic; no empirical result claimed"
    },
    {
      "check_id": "A-S08-05",
      "cluster": "A",
      "article_id": "S08",
      "article_location": "A/articles/ai-latency-cost-per-successful-task.md:57 (Calculate cost per accepted task)",
      "claim": "$1.20/80 and $1.50/90 yield $0.015 and about $0.0167.",
      "source_url": "",
      "source_title": "Reviewer arithmetic on labelled hypothetical inputs",
      "verified_evidence": "1.20/80=.015; 1.50/90=.0166667. Separate labelled hypothetical run examples.",
      "verdict": "supported",
      "severity": "none",
      "accessed_on": "2026-10-22",
      "verification_method": "Independent arithmetic; no empirical result claimed"
    },
    {
      "check_id": "A-S09-01",
      "cluster": "A",
      "article_id": "S09",
      "article_location": "A/articles/evaluate-ai-agents-tools-outcomes.md:39 (Check outcomes and permissions separately)",
      "claim": "τ-bench compares final database state with annotated goal state using tools, policies and simulated users.",
      "source_url": "https://arxiv.org/html/2406.12045v1#S3",
      "source_title": "τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains",
      "verified_evidence": "Abstract and section 3 describe these components and state-based reward.",
      "verdict": "supported",
      "severity": "none",
      "accessed_on": "2026-10-22",
      "verification_method": "Primary page/paper inspected",
      "citation_status": "cited_in_article"
    },
    {
      "check_id": "A-S09-02",
      "cluster": "A",
      "article_id": "S09",
      "article_location": "A/articles/evaluate-ai-agents-tools-outcomes.md:39 (Check outcomes and permissions separately)",
      "claim": "Final-state success alone cannot establish all permission/confirmation constraints.",
      "source_url": "https://arxiv.org/html/2406.12045v1#S3",
      "source_title": "τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains",
      "verified_evidence": "Section 3 explicitly gives missing confirmation as a policy violation that can coexist with reward 1.",
      "verdict": "supported",
      "severity": "none",
      "accessed_on": "2026-10-22",
      "verification_method": "Primary page/paper inspected",
      "citation_status": "cited_in_article"
    },
    {
      "check_id": "A-S09-03",
      "cluster": "A",
      "article_id": "S09",
      "article_location": "A/articles/evaluate-ai-agents-tools-outcomes.md:69 (Measure consistency with the right metric)",
      "claim": "τ-bench pass^k requires all k i.i.d. trials to succeed; pass@k requires at least one.",
      "source_url": "https://arxiv.org/html/2406.12045v1#S3",
      "source_title": "τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains",
      "verified_evidence": "Section 3 defines both and averages across tasks; article correctly avoids exponentiating the aggregate mean.",
      "verdict": "supported",
      "severity": "none",
      "accessed_on": "2026-10-22",
      "verification_method": "Primary page/paper inspected",
      "citation_status": "cited_in_article"
    },
    {
      "check_id": "A-S10-01",
      "cluster": "A",
      "article_id": "S10",
      "article_location": "A/articles/maintain-llm-evaluation-suite.md:17 (Give the evidence an owner)",
      "claim": "OpenAI recommends ongoing evaluation, monitoring new cases and growing the suite.",
      "source_url": "https://developers.openai.com/api/docs/guides/evaluation-best-practices",
      "source_title": "Evaluation best practices | OpenAI API",
      "verified_evidence": "Evaluation-process step 5 explicitly recommends continuous runs and expansion from monitored cases.",
      "verdict": "supported",
      "severity": "none",
      "accessed_on": "2026-10-22",
      "verification_method": "Primary page/paper inspected",
      "citation_status": "cited_in_article"
    },
    {
      "check_id": "A-S10-02",
      "cluster": "A",
      "article_id": "S10",
      "article_location": "A/articles/maintain-llm-evaluation-suite.md:37 (Use layers with different jobs)",
      "claim": "A developer-visible regression set is weaker evidence of fresh generalisation after repeated tuning.",
      "source_url": "https://scikit-learn.org/stable/common_pitfalls.html",
      "source_title": "Common pitfalls and recommended practices",
      "verified_evidence": "Test data must not drive model selection; applying this to exposed prompt-development cases is sound editorial reasoning.",
      "verdict": "supported",
      "severity": "none",
      "accessed_on": "2026-10-22",
      "verification_method": "Primary page/paper inspected",
      "citation_status": "independent corroborating source; not a nearby article citation"
    },
    {
      "check_id": "B-model-benchmarking-01",
      "cluster": "B",
      "article_id": "model-benchmarking",
      "article_location": "B/articles/choosing-model-benchmark.md:31 (Use public benchmarks as a map, not a verdict)",
      "claim": "MMLU is a 57-subject multiple-choice academic/professional test.",
      "source_url": "https://arxiv.org/abs/2009.03300",
      "source_title": "Measuring Massive Multitask Language Understanding",
      "verified_evidence": "Original abstract gives 57-subject scope; full-text section 3 explicitly describes multiple-choice questions.",
      "verdict": "supported",
      "severity": "none",
      "accessed_on": "2026-10-22",
      "verification_method": "Primary page/paper inspected",
      "citation_status": "cited_in_article",
      "inspected_url": "https://arxiv.org/html/2009.03300v3"
    },
    {
      "check_id": "B-model-benchmarking-02",
      "cluster": "B",
      "article_id": "model-benchmarking",
      "article_location": "B/articles/choosing-model-benchmark.md:31 (Use public benchmarks as a map, not a verdict)",
      "claim": "HELM uses multiple scenarios and seven named dimensions.",
      "source_url": "https://arxiv.org/abs/2211.09110",
      "source_title": "Holistic Evaluation of Language Models",
      "verified_evidence": "Original abstract lists the seven dimensions named in the article.",
      "verdict": "supported",
      "severity": "none",
      "accessed_on": "2026-10-22",
      "verification_method": "Primary page/paper inspected",
      "citation_status": "cited_in_article"
    },
    {
      "check_id": "B-model-benchmarking-03",
      "cluster": "B",
      "article_id": "model-benchmarking",
      "article_location": "B/articles/choosing-model-benchmark.md:74 (Put cost and latency into the same decision)",
      "claim": "Pricing may include several token categories and tool calls.",
      "source_url": "https://developers.openai.com/api/docs/pricing",
      "source_title": "Pricing | OpenAI API",
      "verified_evidence": "Official category headers and tool table confirm this; no live rate is reproduced.",
      "verdict": "supported",
      "severity": "none",
      "accessed_on": "2026-10-22",
      "verification_method": "Primary page/paper inspected",
      "citation_status": "cited_in_article"
    },
    {
      "check_id": "B-model-benchmarking-04",
      "cluster": "B",
      "article_id": "model-benchmarking",
      "article_location": "B/articles/choosing-model-benchmark.md:80 (Evaluate an agent as a system)",
      "claim": "SWE-bench pairs real repository issues with codebases and tests code changes.",
      "source_url": "https://arxiv.org/abs/2310.06770",
      "source_title": "SWE-bench: Can Language Models Resolve Real-World GitHub Issues?",
      "verified_evidence": "Original abstract describes GitHub issues, repositories and required code edits; article does not quote historical scores.",
      "verdict": "supported",
      "severity": "none",
      "accessed_on": "2026-10-22",
      "verification_method": "Primary page/paper inspected",
      "citation_status": "cited_in_article"
    },
    {
      "check_id": "B-task-evaluation-01",
      "cluster": "B",
      "article_id": "task-evaluation",
      "article_location": "B/articles/build-task-eval.md:45 (Separate development from final evaluation)",
      "claim": "Splits must keep related incident/template examples together for unseen-group generalisation.",
      "source_url": "https://scikit-learn.org/stable/modules/cross_validation.html",
      "source_title": "Cross-validation: evaluating estimator performance",
      "verified_evidence": "Grouped-data documentation explains dependent samples and separation of groups.",
      "verdict": "supported",
      "severity": "none",
      "accessed_on": "2026-10-22",
      "verification_method": "Primary page/paper inspected",
      "citation_status": "independent corroborating source; not a nearby article citation"
    },
    {
      "check_id": "B-task-evaluation-02",
      "cluster": "B",
      "article_id": "task-evaluation",
      "article_location": "B/articles/build-task-eval.md:45 (Separate development from final evaluation)",
      "claim": "Inspecting test failures to tune configuration compromises untouched test status.",
      "source_url": "https://scikit-learn.org/stable/common_pitfalls.html",
      "source_title": "Common pitfalls and recommended practices",
      "verified_evidence": "Official leakage guidance prohibits test-driven model choices; prompt tuning is an analogous application.",
      "verdict": "supported",
      "severity": "none",
      "accessed_on": "2026-10-22",
      "verification_method": "Primary page/paper inspected",
      "citation_status": "independent corroborating source; not a nearby article citation"
    },
    {
      "check_id": "B-task-evaluation-03",
      "cluster": "B",
      "article_id": "task-evaluation",
      "article_location": "B/articles/build-task-eval.md:69 (Example: a small first version)",
      "claim": "Illustrative 120-case suite comprises 70+25+25, with 30 held out and 90 development cases.",
      "source_url": "",
      "source_title": "Reviewer arithmetic on labelled hypothetical inputs",
      "verified_evidence": "70+25+25=120 and 120-30=90. Example explicitly disclaims universal sample-size advice; weighting still needs specification.",
      "verdict": "supported",
      "severity": "none",
      "accessed_on": "2026-10-22",
      "verification_method": "Independent arithmetic; no empirical result claimed"
    },
    {
      "check_id": "B-dataset-integrity-01",
      "cluster": "B",
      "article_id": "dataset-integrity",
      "article_location": "B/articles/datasets-holdouts-contamination.md:17 (Name the kind of generalisation you need)",
      "claim": "Random row splitting can be misleading for dependent groups.",
      "source_url": "https://scikit-learn.org/stable/modules/cross_validation.html",
      "source_title": "Cross-validation: evaluating estimator performance",
      "verified_evidence": "Official grouped-split guidance supports this principle; local examples are recommendations.",
      "verdict": "supported",
      "severity": "none",
      "accessed_on": "2026-10-22",
      "verification_method": "Primary page/paper inspected",
      "citation_status": "independent corroborating source; not a nearby article citation"
    },
    {
      "check_id": "B-dataset-integrity-02",
      "cluster": "B",
      "article_id": "dataset-integrity",
      "article_location": "B/articles/datasets-holdouts-contamination.md:25 (Understand different forms of leakage)",
      "claim": "Yang et al. propose comparing original and rewritten benchmark items to probe memorisation.",
      "source_url": "https://arxiv.org/abs/2311.04850",
      "source_title": "Rethinking Benchmark and Contamination for Language Models with Rephrased Samples",
      "verified_evidence": "Full text sections 3–5 principally constructs paraphrased training contamination and tests decontamination. It supports paraphrase-overlap risk, but not the described original-versus-rewrite inference protocol.",
      "verdict": "partially_supported",
      "severity": "major",
      "accessed_on": "2026-10-22",
      "verification_method": "Primary page/paper inspected",
      "inspected_url": "https://arxiv.org/html/2311.04850v2",
      "citation_classification": "Real relevant source; method characterised imprecisely, not a fictional citation.",
      "citation_status": "cited_in_article"
    },
    {
      "check_id": "B-dataset-integrity-03",
      "cluster": "B",
      "article_id": "dataset-integrity",
      "article_location": "B/articles/datasets-holdouts-contamination.md:25 (Understand different forms of leakage)",
      "claim": "Xiao and Cheng’s 2026 preprint reports score inflation without much relative reordering in its studied benchmarks.",
      "source_url": "https://arxiv.org/abs/2609.02899",
      "source_title": "Contamination Inflates Scores but Rarely Reorders Large Language Model Leaderboards",
      "verified_evidence": "Title, authors and abstract match; abstract reports a paraphrase-controlled ranking analysis with limited benchmark/model scope.",
      "verdict": "supported",
      "severity": "none",
      "accessed_on": "2026-10-22",
      "verification_method": "Primary page/paper inspected",
      "citation_status": "cited_in_article"
    },
    {
      "check_id": "B-evaluation-metrics-01",
      "cluster": "B",
      "article_id": "evaluation-metrics",
      "article_location": "B/articles/metrics-rubrics.md:17 (Start with observable outcomes)",
      "claim": "Precision measures correct predicted positives; recall measures found actual positives.",
      "source_url": "https://scikit-learn.org/stable/modules/generated/sklearn.metrics.precision_recall_fscore_support.html",
      "source_title": "precision_recall_fscore_support",
      "verified_evidence": "Official formulas confirm these uncited definitions.",
      "verdict": "supported",
      "severity": "none",
      "accessed_on": "2026-10-22",
      "verification_method": "Primary page/paper inspected",
      "citation_status": "independent corroborating source; not a nearby article citation"
    },
    {
      "check_id": "B-evaluation-metrics-02",
      "cluster": "B",
      "article_id": "evaluation-metrics",
      "article_location": "B/articles/metrics-rubrics.md:25 (Separate dimensions before combining them)",
      "claim": "OpenAI warns against generic academic metrics alone and favours task-specific evaluation.",
      "source_url": "https://developers.openai.com/api/docs/guides/evaluation-best-practices",
      "source_title": "Evaluation best practices | OpenAI API",
      "verified_evidence": "Best-practice and anti-pattern bullets support this scoped attribution.",
      "verdict": "supported",
      "severity": "none",
      "accessed_on": "2026-10-22",
      "verification_method": "Primary page/paper inspected",
      "citation_status": "cited_in_article"
    },
    {
      "check_id": "B-evaluation-metrics-03",
      "cluster": "B",
      "article_id": "evaluation-metrics",
      "article_location": "B/articles/metrics-rubrics.md:41 (Validate automated metrics)",
      "claim": "OpenAI grader guidance covers string/similarity/code/model grading and tests against known quality.",
      "source_url": "https://developers.openai.com/api/docs/guides/graders",
      "source_title": "Graders | OpenAI API",
      "verified_evidence": "Grader types and best-practice section describe these methods and ground-truth-grade validation.",
      "verdict": "supported",
      "severity": "none",
      "accessed_on": "2026-10-22",
      "verification_method": "Primary page/paper inspected",
      "citation_status": "cited_in_article"
    },
    {
      "check_id": "B-evaluation-metrics-04",
      "cluster": "B",
      "article_id": "evaluation-metrics",
      "article_location": "B/articles/metrics-rubrics.md:17 (Start with observable outcomes)",
      "claim": "Always predicting routine on a 95%-routine set misses all urgent cases despite 95% accuracy.",
      "source_url": "",
      "source_title": "Reviewer arithmetic on labelled hypothetical inputs",
      "verified_evidence": "For 100 illustrative items: TN=95, FN=5, TP=FP=0, so accuracy=.95 and urgent recall=0.",
      "verdict": "supported",
      "severity": "none",
      "accessed_on": "2026-10-22",
      "verification_method": "Independent arithmetic; no empirical result claimed"
    },
    {
      "check_id": "B-human-judging-01",
      "cluster": "B",
      "article_id": "human-judging",
      "article_location": "B/articles/human-evaluation-model-judges.md:31 (Reduce avoidable bias)",
      "claim": "Position bias and order-dependent preferences are documented, attributed here to Zheng et al. 2024.",
      "source_url": "https://arxiv.org/abs/2406.07791",
      "source_title": "Judging the Judges: A Systematic Study of Position Bias in LLM-as-a-Judge",
      "verified_evidence": "Linked paper supports position bias but lists Lin Shi et al., not Zheng; both inline occurrences require author correction.",
      "verdict": "partially_supported",
      "severity": "minor",
      "accessed_on": "2026-10-22",
      "verification_method": "Primary page/paper inspected",
      "citation_classification": "Real source and supported phenomenon; mismatched author attribution.",
      "citation_status": "cited_in_article"
    },
    {
      "check_id": "B-human-judging-02",
      "cluster": "B",
      "article_id": "human-judging",
      "article_location": "B/articles/human-evaluation-model-judges.md:37 (Use model judges as instruments)",
      "claim": "OpenAI recommends specific criteria/examples and human-feedback calibration of model grading.",
      "source_url": "https://developers.openai.com/api/docs/guides/evaluation-best-practices",
      "source_title": "Evaluation best practices | OpenAI API",
      "verified_evidence": "Calibration and criterion-specific comparison guidance support this attribution.",
      "verdict": "supported",
      "severity": "none",
      "accessed_on": "2026-10-22",
      "verification_method": "Primary page/paper inspected",
      "citation_status": "cited_in_article"
    },
    {
      "check_id": "B-human-judging-03",
      "cluster": "B",
      "article_id": "human-judging",
      "article_location": "B/articles/human-evaluation-model-judges.md:37 (Use model judges as instruments)",
      "claim": "Model judges can have verbosity and self-preference biases.",
      "source_url": "https://arxiv.org/abs/2306.05685",
      "source_title": "Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena",
      "verified_evidence": "MT-Bench paper documents those effects; B gives no nearby primary citation for this additional factual claim.",
      "verdict": "supported",
      "severity": "none",
      "accessed_on": "2026-10-22",
      "verification_method": "Primary page/paper inspected",
      "citation_status": "independent corroborating source; not a nearby article citation"
    },
    {
      "check_id": "B-repeatability-01",
      "cluster": "B",
      "article_id": "repeatability",
      "article_location": "B/articles/repeatability-uncertainty.md:40 (Report intervals with their assumptions)",
      "claim": "Bouthillier et al. discuss variance, intervals and performance-comparison tests.",
      "source_url": "https://arxiv.org/abs/2103.03098",
      "source_title": "Accounting for Variance in Machine Learning Benchmarks",
      "verified_evidence": "Abstract and full text sections 4 and appendix C cover these topics; no result is transplanted to current API models.",
      "verdict": "supported",
      "severity": "none",
      "accessed_on": "2026-10-22",
      "verification_method": "Primary page/paper inspected",
      "inspected_url": "https://arxiv.org/html/2103.03098v1",
      "citation_status": "cited_in_article"
    },
    {
      "check_id": "B-repeatability-02",
      "cluster": "B",
      "article_id": "repeatability",
      "article_location": "B/articles/repeatability-uncertainty.md:34 (Compare candidates on paired cases)",
      "claim": "Pairing preserves shared task conditions and can reduce case-difficulty noise.",
      "source_url": "https://arxiv.org/abs/2103.03098",
      "source_title": "Accounting for Variance in Machine Learning Benchmarks",
      "verified_evidence": "Appendix C.2 discusses shared random factors and variance reduction in paired comparisons.",
      "verdict": "supported",
      "severity": "none",
      "accessed_on": "2026-10-22",
      "verification_method": "Primary page/paper inspected",
      "citation_status": "cited_in_article"
    },
    {
      "check_id": "B-repeatability-03",
      "cluster": "B",
      "article_id": "repeatability",
      "article_location": "B/articles/repeatability-uncertainty.md:23 (Repeat the right thing)",
      "claim": "100 cases repeated three times produce 300 outputs but only 100 distinct cases.",
      "source_url": "",
      "source_title": "Reviewer arithmetic on labelled hypothetical inputs",
      "verified_evidence": "100*3=300 outputs; three calls do not create new independent case identities.",
      "verdict": "supported",
      "severity": "none",
      "accessed_on": "2026-10-22",
      "verification_method": "Independent arithmetic; no empirical result claimed"
    },
    {
      "check_id": "B-fair-comparison-01",
      "cluster": "B",
      "article_id": "fair-comparison",
      "article_location": "B/articles/fair-model-comparisons.md:23 (Freeze the surrounding system)",
      "claim": "Identical literal prompt text need not create equivalent conditions across model families.",
      "source_url": "https://arxiv.org/abs/2310.11324",
      "source_title": "Quantifying Language Models' Sensitivity to Spurious Features in Prompt Design or: How I learned to start worrying about prompt formatting",
      "verified_evidence": "Original format-sensitivity study finds weak cross-model transfer; article’s caution is reasonable but lacks a nearby factual citation.",
      "verdict": "supported",
      "severity": "none",
      "accessed_on": "2026-10-22",
      "verification_method": "Primary page/paper inspected",
      "citation_status": "independent corroborating source; not a nearby article citation"
    },
    {
      "check_id": "B-fair-comparison-02",
      "cluster": "B",
      "article_id": "fair-comparison",
      "article_location": "B/articles/fair-model-comparisons.md:37 (Make the runtime comparable)",
      "claim": "A stream’s first output and completed usable output are different latency boundaries.",
      "source_url": "https://developers.openai.com/api/docs/guides/latency-optimization",
      "source_title": "Latency optimization",
      "verified_evidence": "Streaming guidance distinguishes waiting/progress from full completion; the article’s measurement recommendation is sound.",
      "verdict": "supported",
      "severity": "none",
      "accessed_on": "2026-10-22",
      "verification_method": "Primary page/paper inspected",
      "citation_status": "independent corroborating source; not a nearby article citation"
    },
    {
      "check_id": "B-latency-cost-01",
      "cluster": "B",
      "article_id": "latency-cost",
      "article_location": "B/articles/latency-cost.md:23 (Calculate cost transparently)",
      "claim": "Official pricing distinguishes token categories and charges for some tools.",
      "source_url": "https://developers.openai.com/api/docs/pricing",
      "source_title": "Pricing | OpenAI API",
      "verified_evidence": "Pricing and tool tables confirm these categories.",
      "verdict": "supported",
      "severity": "none",
      "accessed_on": "2026-10-22",
      "verification_method": "Primary page/paper inspected",
      "citation_status": "cited_in_article"
    },
    {
      "check_id": "B-latency-cost-02",
      "cluster": "B",
      "article_id": "latency-cost",
      "article_location": "B/articles/latency-cost.md:35 (Measure latency users feel)",
      "claim": "Streaming first token can precede a completed actionable answer.",
      "source_url": "https://developers.openai.com/api/docs/guides/latency-optimization",
      "source_title": "Latency optimization",
      "verified_evidence": "Official streaming discussion supports measuring visible progress separately from completion.",
      "verdict": "supported",
      "severity": "none",
      "accessed_on": "2026-10-22",
      "verification_method": "Primary page/paper inspected",
      "citation_status": "independent corroborating source; not a nearby article citation"
    },
    {
      "check_id": "B-latency-cost-03",
      "cluster": "B",
      "article_id": "latency-cost",
      "article_location": "B/articles/latency-cost.md:55 (Example worksheet)",
      "claim": "A costs $258 and B $127 after review at $1/minute for 60 and 25 four-minute cases.",
      "source_url": "",
      "source_title": "Reviewer arithmetic on labelled hypothetical inputs",
      "verified_evidence": "A=18+60*4=258; B=27+25*4=127. If all repairs succeed, denominators are 500: .516 and .254 per completed case; article stops before showing these ratios.",
      "verdict": "supported",
      "severity": "none",
      "accessed_on": "2026-10-22",
      "verification_method": "Independent arithmetic; no empirical result claimed"
    },
    {
      "check_id": "B-agent-evaluation-01",
      "cluster": "B",
      "article_id": "agent-evaluation",
      "article_location": "B/articles/agent-system-evaluation.md:31 (Evaluate trajectories as well as outcomes)",
      "claim": "Official agent-eval guide recommends traces for tool choice, handoffs and workflow questions.",
      "source_url": "https://developers.openai.com/api/docs/guides/agent-evals",
      "source_title": "Evaluate agent workflows | OpenAI API",
      "verified_evidence": "Trace-grading section describes tool/handoff/policy assessment.",
      "verdict": "supported",
      "severity": "none",
      "accessed_on": "2026-10-22",
      "verification_method": "Primary page/paper inspected",
      "citation_status": "cited_in_article"
    },
    {
      "check_id": "B-agent-evaluation-02",
      "cluster": "B",
      "article_id": "agent-evaluation",
      "article_location": "B/articles/agent-system-evaluation.md:37 (Control the environment)",
      "claim": "2026 SWE-Bench Pro Verified preprint reports solution/test leakage and task-quality concerns.",
      "source_url": "https://arxiv.org/abs/2609.08149",
      "source_title": "SWE-Bench Pro Verified: A Reliable Benchmark for Software Engineering Agents",
      "verified_evidence": "Matching title/authors; abstract names reward hacking, leaked solutions/evaluation information and flawed tasks; article retains benchmark-specific caveat.",
      "verdict": "supported",
      "severity": "none",
      "accessed_on": "2026-10-22",
      "verification_method": "Primary page/paper inspected",
      "citation_status": "cited_in_article"
    },
    {
      "check_id": "B-evaluation-maintenance-01",
      "cluster": "B",
      "article_id": "evaluation-maintenance",
      "article_location": "B/articles/maintaining-evaluations.md:23 (Build a continuous evaluation loop)",
      "claim": "OpenAI recommends continuous evaluation on changes and new cases from nondeterminism/failures.",
      "source_url": "https://developers.openai.com/api/docs/guides/evaluation-best-practices",
      "source_title": "Evaluation best practices | OpenAI API",
      "verified_evidence": "Process step 5 supports the attribution.",
      "verdict": "supported",
      "severity": "none",
      "accessed_on": "2026-10-22",
      "verification_method": "Primary page/paper inspected",
      "citation_status": "cited_in_article"
    },
    {
      "check_id": "B-evaluation-maintenance-02",
      "cluster": "B",
      "article_id": "evaluation-maintenance",
      "article_location": "B/articles/maintaining-evaluations.md:23 (Build a continuous evaluation loop)",
      "claim": "An exposed regression suite and protected holdout support different generalisation claims.",
      "source_url": "https://scikit-learn.org/stable/common_pitfalls.html",
      "source_title": "Common pitfalls and recommended practices",
      "verified_evidence": "Official prohibition on test-driven choices supports the separation; specific pipeline is editorial advice.",
      "verdict": "supported",
      "severity": "none",
      "accessed_on": "2026-10-22",
      "verification_method": "Primary page/paper inspected",
      "citation_status": "independent corroborating source; not a nearby article citation"
    },
    {
      "check_id": "B-public-benchmarks-01",
      "cluster": "B",
      "article_id": "public-benchmarks",
      "article_location": "B/articles/public-benchmark-literacy.md:17 (Ask what the benchmark measures)",
      "claim": "MMLU’s original scope is 57 academic/professional multiple-choice subjects.",
      "source_url": "https://arxiv.org/abs/2009.03300",
      "source_title": "Measuring Massive Multitask Language Understanding",
      "verified_evidence": "Original abstract and task description support the scoped description.",
      "verdict": "supported",
      "severity": "none",
      "accessed_on": "2026-10-22",
      "verification_method": "Primary page/paper inspected",
      "citation_status": "cited_in_article",
      "inspected_url": "https://arxiv.org/html/2009.03300v3"
    },
    {
      "check_id": "B-public-benchmarks-02",
      "cluster": "B",
      "article_id": "public-benchmarks",
      "article_location": "B/articles/public-benchmark-literacy.md:25 (Check whether results are comparable)",
      "claim": "HELM combines scenarios with dimensions beyond accuracy.",
      "source_url": "https://arxiv.org/abs/2211.09110",
      "source_title": "Holistic Evaluation of Language Models",
      "verified_evidence": "Original abstract confirms its multi-metric framework.",
      "verdict": "supported",
      "severity": "none",
      "accessed_on": "2026-10-22",
      "verification_method": "Primary page/paper inspected",
      "citation_status": "cited_in_article"
    },
    {
      "check_id": "B-public-benchmarks-03",
      "cluster": "B",
      "article_id": "public-benchmarks",
      "article_location": "B/articles/public-benchmark-literacy.md:33 (Understand the score, not just the rank)",
      "claim": "Position bias attribution says Zheng et al. 2024.",
      "source_url": "https://arxiv.org/abs/2406.07791",
      "source_title": "Judging the Judges: A Systematic Study of Position Bias in LLM-as-a-Judge",
      "verified_evidence": "URL resolves to the relevant position-bias study by Lin Shi et al.; phenomenon supported, author label wrong.",
      "verdict": "partially_supported",
      "severity": "minor",
      "accessed_on": "2026-10-22",
      "verification_method": "Primary page/paper inspected",
      "citation_classification": "Mismatched author attribution; no fabricated paper.",
      "citation_status": "cited_in_article"
    },
    {
      "check_id": "B-public-benchmarks-04",
      "cluster": "B",
      "article_id": "public-benchmarks",
      "article_location": "B/articles/public-benchmark-literacy.md:41 (Consider contamination and task exposure)",
      "claim": "Rephrased-samples study is presented as probing reliance on remembered surface forms.",
      "source_url": "https://arxiv.org/abs/2311.04850",
      "source_title": "Rethinking Benchmark and Contamination for Language Models with Rephrased Samples",
      "verified_evidence": "Full text instead emphasises paraphrased contamination evading overlap filters. Revise the paraphrase-risk summary or cite an actual original-versus-rewrite probe.",
      "verdict": "partially_supported",
      "severity": "major",
      "accessed_on": "2026-10-22",
      "verification_method": "Primary page/paper inspected",
      "inspected_url": "https://arxiv.org/html/2311.04850v2",
      "citation_classification": "Real relevant source; unsupported specificity in method summary.",
      "citation_status": "cited_in_article"
    },
    {
      "check_id": "B-repeatability-04",
      "cluster": "B",
      "article_id": "repeatability",
      "article_location": "B/articles/repeatability-uncertainty.md:17 (Name the sources of variation)",
      "claim": "Fixed seed need not guarantee identical outputs across backend configurations.",
      "source_url": "https://developers.openai.com/cookbook/examples/reproducible_outputs_with_the_seed_parameter",
      "source_title": "How to make your completions outputs consistent with the new seed parameter",
      "verified_evidence": "Official historical recipe describes best-effort determinism and backend fingerprint changes. Supports the qualified principle, not availability of a seed control on every current model.",
      "verdict": "supported",
      "severity": "none",
      "accessed_on": "2026-10-22",
      "citation_status": "independent corroboration; no nearby article citation",
      "verification_method": "Primary passage inspected"
    },
    {
      "check_id": "B-latency-cost-04",
      "cluster": "B",
      "article_id": "latency-cost",
      "article_location": "B/articles/latency-cost.md:23 (Calculate cost transparently)",
      "claim": "Models can tokenise identical text differently.",
      "source_url": "https://developers.openai.com/cookbook/examples/how_to_count_tokens_with_tiktoken",
      "source_title": "How to count tokens with Tiktoken",
      "verified_evidence": "Official archived recipe shows different encodings and different splits/counts for identical text. Only the tokenisation principle is used, not its old model catalogue.",
      "verdict": "supported",
      "severity": "none",
      "accessed_on": "2026-10-22",
      "citation_status": "independent corroboration; no nearby article citation",
      "verification_method": "Primary passage inspected"
    },
    {
      "check_id": "B-human-judging-04",
      "cluster": "B",
      "article_id": "human-judging",
      "article_location": "B/articles/human-evaluation-model-judges.md:53 (Measure agreement and inspect what it means)",
      "claim": "Long batches can reduce reviewer attention; fatigue should be monitored.",
      "source_url": "",
      "source_title": "No independent human-factors source inspected",
      "verified_evidence": "General process caution was read but not independently verified against a primary human-factors study; no effect size or measured finding is claimed.",
      "verdict": "unverified",
      "severity": "minor",
      "accessed_on": "2026-10-22",
      "citation_status": "uncited; unchecked general assertion",
      "verification_method": "Explicit coverage limitation"
    }
  ],
  "coverage_notes": [
    "All eleven full articles and all supporting files in each cluster were read. Each article has at least two substantive checks; arithmetic checks are calculations on expressly hypothetical inputs, not measured results.",
    "Every distinct external article citation URL was opened, including anchored HumanEval and τ-bench full text. All distinct registered source families were inspected, including the workflow-only Long-running work reference. Source records and mappings were also inspected locally.",
    "Historical source design claims were checked against original paper abstracts when sufficient; full text inspected for HumanEval pass@k, τ-bench definitions/limitations, paraphrased contamination and benchmark variance. Official documentation passages were inspected beyond navigation/search snippets.",
    "Every numerical worked example was recalculated. Illustrative sample counts, budgets, cadences, gates and thresholds are not empirical assertions. No real model experiment, API call, invoice or deployment was performed.",
    "Changing factual claims about SWE-bench views and provider billing/cache/reasoning/streaming were checked on 22 October 2026; no current named-model rank or fixed price quote appears in either cluster.",
    "This is targeted material-claim verification, not exhaustive replication of studies or an audit of every sentence. Uncited procedural recommendations are evaluated editorially. No original study’s empirical findings were independently reproduced.",
    "No cited article source was inaccessible in the final inspection. No fabricated citation or empirical result was established. B has two repeated author mismatches and two imprecise descriptions of a real contamination paper.",
    "Citation-label counting differences and removed completion timestamps are not fabrication. Identity, runtime, cost, production usage and other private files were not accessed.",
    "Explicit unchecked claim: B human-judging’s long-batch/fatigue assertion has no independent primary human-factors verification (B-human-judging-04). No quantitative effect is asserted. Other low-consequence general process cautions were not exhaustively sourced. Historical official seed/tokenisation recipes were inspected only for principles, not present feature availability."
  ],
  "workflow_reference_check": {
    "source_url": "https://learn.chatgpt.com/docs/long-running-work",
    "source_title": "Long-running work",
    "inspected_scope": "Define what done means: outcomes, constraints and verification",
    "finding": "Supports the workflow-context description; neither cluster uses it as model-performance evidence."
  },
  "inspection_date": "2026-10-22"
}