Frontier Log

Benchmark Scores Move After Launch Day

How labs re-report their older models' benchmark scores in later documents, and how far a launch-day score holds.

OpenAI's and Anthropic's later reports that moved an older model's score 2 points or more. OpenAI: 10 of its 17 later reports, 7 lowered and 3 raised. Anthropic: 8 of its 28, 4 lowered and 4 raised. Each as a bar from zero. OpenAI: GPT-6 Astra on AutomationBench −11.1, in the GPT-6 Sol and Luna post; o3 on HLE, no tools −5.6, in the GPT-5 post; GPT-4o on GPQA Diamond −3.0, in the Learning to reason with LLMs post; o3-mini on GPQA Diamond −2.7, in the o3 and o4-mini post; o1-preview on AIME 2024, no tools −2.6, in the o1 for developers post; o3-mini on GPQA Diamond −2.5, in the GPT-4.1 post; GPT-5 on SWE-bench Verified −2.1, in the GPT-5.1 in the API post; o1 on GPQA Diamond +2.3, in the o3 and o4-mini post; GPT-5.6 Sol on OSWorld 2.0, offline set, partial score +3.1, in the GPT-6 Astra post; GPT-6 Astra on TB-Science 0.1 +3.5, in the GPT-6.1 Sol post. Anthropic: Opus 5 on FrontierCode 1.1, main set −5.4, in the Claude Opus 5.5 post; Sonnet 5 on AutomationBench −2.8, in the Claude Sonnet 5.5 system card; Sonnet 5 on HLE, with tools −2.5, in the Claude Sonnet 5.5 post; Sonnet 4.6 on HLE, with tools −2.2, in the Claude Sonnet 5 post; Opus 5.5 on AutomationBench +2.5, in the Claude Sonnet 5.5 system card; Opus 5 on OSWorld 2.0, offline set, partial score +4.8, in the Claude Fable 5.1 and Mythos 5.1 post; Sonnet 4.5 on HLE, with tools +5.2, in the Claude Opus 4.6 post; Opus 4.8 on Terminal-Bench 2.1 +8.1, in the Claude Sonnet 5 post.

OpenAI marks its old models down

7 of OpenAI's 10 large moves were down; Anthropic's split 4 and 4*

Counting moves of 2 points or more, OpenAI's later reports lowered an older model's score 7 times out of 10, while Anthropic's split 4 and 4. OpenAI's biggest is GPT-6 Astra on AutomationBench, 11.1 points lower in the GPT-6 Sol and Luna post, at low effort.

* Counting later reports that moved an older model's score 2 points or more

Later reports that moved an older model's score 2 points or more, any benchmark, by lab, as lowered and raised counts: Meta 3 lowered, 1 raised; OpenAI 7 lowered, 3 raised; Z.ai 2 lowered, 1 raised; Anthropic 4 lowered, 4 raised; NVIDIA 3 lowered, 5 raised; Google 0 lowered, 3 raised; Other labs (Alibaba, DeepSeek, MiniMax, Mistral) 0 lowered, 3 raised.

Re-reported scores go both ways

19 later reports lowered the old score, 20 raised it*

Counting moves of 2 points or more, labs re-report older models' scores in both directions: 19 later reports lowered the old score and 20 raised it. OpenAI lowered 7 of its 10 and Meta 3 of 4; NVIDIA, Google and the smaller labs mostly raised theirs.

* Counting later reports that moved an older model's score 2 points or more

Re-reported launch-day scores by size of the change: under 1 point 26, 1 to 2 points 14, 2 to 5 points 24, 5 or more 13. The two benchmarks that moved most, each with its five largest later reports, as arrows from the first report to the later one. BrowseComp, median move 8.5 points over 7: Nemotron 3 Ultra 550B-A55B 44.4 to 61.06 (+16.7), MiniMax M2 44.0 to 56.9 (+12.9), GLM-4.6 45.1 to 57.5 (+12.4), Nemotron 3 Super 120B-A12B 31.28 to 22.77 (−8.5), Nemotron 3.5 Lightning 30B-A3B 36.97 to 39.50 (+2.5). AutomationBench, median move 2.5 points over 7: DeepSeek-V4-Pro-0813 31.8 to 43.2 (+11.4), GPT-6 Astra 41.4 to 30.3 (−11.1), Sonnet 5 13.5 to 10.7 (−2.8), Opus 5.5 40.0 to 42.5 (+2.5), Opus 4.8 15.5 to 17.0 (+1.5).

How far a launch-day score holds

37 of 77 re-reported launch scores moved 2 points or more

About half the launch-day scores a lab later printed again moved 2 points or more, and 13 moved 5 or more. BrowseComp moves most, a median 8.5 points, as harnesses and context management change.

Six later reports, one for each kind of change, each the first report over the later one with the setup each states: Sonnet 4.6 on HLE, with tools, 49.0 first reported Feb 2026 (web search, web fetch, code execution, programmatic tool calling; compaction at 50K up to 3M tokens; max effort), 46.8 in a later report Jun 2026 (re-graded with an updated grader model), −2.2, grading changed; Nemotron 3 Ultra 550B-A55B on BrowseComp, 44.4 first reported Jun 2026 (internal search harness), 61.06 in a later report Oct 2026 (OpenCode harness), +16.7, different harness; MiniMax M2 on BrowseComp, 44.0 first reported Oct 2025 (setup not stated), 56.9 in a later report Dec 2025 (with context management), +12.9, context management added; GPT-6 Astra on AutomationBench, 41.4 first reported Sep 2026 (setup not stated), 30.3 in a later report Sep 2026 (low effort; AutomationBench 1.0.6), −11.1, different effort; o3 on HLE, no tools, 20.32 first reported Apr 2025 (high reasoning effort), 14.7 in a later report Aug 2025 (rerun on a later version of HLE; full set), −5.6, different task set or version; DeepSeek-V4-Pro-0813 on AutomationBench, 31.8 first reported Aug 2026 (public set), 43.2 in a later report Sep 2026 (set not stated), +11.4, no change stated.

What moves a re-reported score

6 later reports, one for each kind of change

A stated change in setup, from the grader to the harness, context management, effort or task set, can move a re-reported score by 2 to 17 points. DeepSeek's AutomationBench later report, 11.4 points higher, states no change.

AIME 2024, no tools

ModelScoreSources
OpenAI o1First report, Sep 2024. Setup: AIME year and pass@1 not stated44.6%Announcement'Optimized for STEM Reasoning' section, the 'Mathematics:' paragraph under the 'Math Performance vs Inference Cost' chart, the figure in parentheses after 'o1-preview'
OpenAI o1Later report, 17 Dec 2024, in the o1 for developers post: no change stated. Setup: pass@142.0%Announcementeval table (Category / Eval / o1-2024-12-17 / o1-preview), row 'Math: AIME 2024 (pass @1)', column 'o1-preview'

AIME 2025, no tools

ModelScoreSources
Llama NemotronFirst report, Mar 2025. Setup: reasoning on; average of up to 16 runs; 32K sequence length58.4%Announcement'Evaluation Results' section, 'AIME25' table, row 'Reasoning On', column 'pass@1'
Llama NemotronLater report, 29 Jul 2025, in the Llama Nemotron Super v1.5 post: no change stated53.3%AnnouncementFigure 2 bar chart (image), value label of the 'Llama Nemotron Super 49B v1' bar (lightest, rightmost in group) in group 'AIME25'
Llama NemotronFirst report, Mar 2025. Setup: reasoning on; average of up to 16 runs; 32K sequence length58.4%Announcement'Evaluation Results' section, 'AIME25' table, row 'Reasoning On', column 'pass@1'
Llama NemotronLater report, 5 Mar 2026, in the NVFP4 quantization-aware distillation report: different eval tools. Setup: BF16 baseline of an NVFP4 quantization study; eval tools differ from the model card; average of 48 runs46.0%AnnouncementTable 2 (SFT-heavy models, PDF page 5), model 'Llama Nemotron Super V1', row 'BF16', column 'AIME25'
Gemini 2.5 ProFirst report, Mar 2025. Setup: pass@1, averaged over multiple trials86.7%Announcementbenchmark table image (gemini_benchmarks_cropped GIF), row 'Mathematics AIME 2025', sub-row 'single attempt (pass@1)', column 'Gemini 2.5 Pro Experimental (03-25)'
Gemini 2.5 ProLater report, 18 Nov 2025, in the Start building with Gemini 3 post: no change stated88.0%Announcementbenchmark table (image), row 'AIME 2025', sub-row 'No tools', column 'Gemini 2.5 Pro'

AutomationBench

ModelScoreSources
Claude Opus 4.8First report, May 2026. Setup: Zapier leaderboard, private held-out set; max effort15.5%AnnouncementSection 8.1, Table 8.1.A (evaluation summary), row 'Automation Bench', column 'Claude Opus 4.8'
Claude Opus 4.8Later report, 24 Jul 2026, in the Claude Opus 5 post: no change stated. Setup: Zapier leaderboard, private held-out set; max effort17.0%Announcementbenchmark table (image), row 'Business workflows, AutomationBench', column 'Opus 4.8'
Claude Fable 5 and Mythos 5First report, Jun 2026. Setup: Zapier leaderboard, private held-out set; max effort17.4%AnnouncementSection 8.1, Table 8.1.A, row 'AutomationBench', column 'Fable 5'
Claude Fable 5 and Mythos 5Later report, 1 Sep 2026, in the Claude Fable 5.1 and Mythos 5.1 post: grading changed. Setup: Zapier leaderboard, private held-out set; tasks blocked by safeguards scored zero17.1%Announcementbenchmark table (HTML), row 'Business workflows, AutomationBench', column 'Fable 5'
Claude Sonnet 5First report, Jun 2026. Setup: Zapier leaderboard, private held-out set; max effort13.5%AnnouncementSection 8.1, Table 8.1.A, row 'AutomationBench', column 'Claude Sonnet 5'
Claude Sonnet 5Later report, 28 Sep 2026, in the Claude Sonnet 5.5 system card: different task set or version. Setup: run by Zapier; private set, release 1.0.610.7%AnnouncementSection 8.1, Table 8.1.A, row 'AutomationBench', column 'Claude Sonnet 5'
Claude Opus 5First report, Jul 2026. Setup: max effort; Zapier leaderboard, private held-out set26.0%Announcementbenchmark table (image), row 'Business workflows, AutomationBench', column 'Opus 5'
Claude Opus 5Later report, 1 Sep 2026, in the Claude Fable 5.1 and Mythos 5.1 post: no change stated. Setup: Zapier leaderboard, private held-out set; max effort26.9%Announcementbenchmark table (HTML), row 'Business workflows, AutomationBench', column 'Opus 5'
DeepSeek-V4-Pro GAFirst report, Aug 2026. Setup: public set31.8%Announcementtable image, row 'AutomationBench (Public)', column 'DeepSeek-V4-Pro-0813'
DeepSeek-V4-Pro GALater report, 10 Sep 2026, in the DeepSeek-V4.1-Flash post: no change stated. Setup: set not stated43.2%Announcementlater report: DeepSeek-V4.1-Flash post, benchmark table image (second image), row 'Automation-Bench', column 'DeepSeek V4-Pro 0813'
GPT-6 AstraFirst report, Sep 202641.4%AnnouncementProfessional table, row 'AutomationBench', column 'GPT‑6 Astra'
GPT-6 AstraLater report, 22 Sep 2026, in the GPT-6 Sol and Luna post: different effort. Setup: low effort; AutomationBench 1.0.630.3%AnnouncementAutomationBench table (Model (and effort) / Score / Cost per task), row 'GPT-6 Astra (low)', column 'Score'
Claude Opus 5.5First report, Sep 2026. Setup: run by Zapier, without fallback models40.0%Announcementbenchmark table, row 'AutomationBench', column 'Opus 5.5'
Claude Opus 5.5Later report, 28 Sep 2026, in the Claude Sonnet 5.5 system card: grading changed. Setup: run by Zapier; refused tasks re-run with default fallbacks; private set, release 1.0.642.5%AnnouncementSection 8.1, Table 8.1.A, row 'AutomationBench', column 'Claude Opus 5.5'

BrowseComp

ModelScoreSources
GLM-4.6First report, Sep 2025. Setup: 128K context45.1%Announcementbenchmark chart image (eight panels), panel 'BrowseComp', first bar 'GLM-4.6', value label
GLM-4.6Later report, 22 Dec 2025, in the GLM-4.7 post: context management added. Setup: with context management57.5%Announcementbenchmark table, row 'BrowseComp w/ Context Manage', column 'GLM-4.6'
MiniMax M2First report, Oct 202544.0%Announcementfirst benchmark chart image (wide, eight panels), panel 'BrowseComp', first bar 'MiniMax-M2' (red), value label
MiniMax M2Later report, 23 Dec 2025, in the MiniMax M2.1 post: context management added. Setup: with context management56.9%Announcement'Benchmarks' section, last (long-horizon tool use and intelligence) table image, row 'BrowseComp (context management)', column 'MiniMax-M2'
Claude Sonnet 4.6First report, Feb 2026. Setup: web search, web fetch, programmatic tool calling; compaction at 50K up to 10M tokens; max effort, no thinking74.7%Announcementbenchmark table (image), row 'Agentic search, BrowseComp', column 'Sonnet 4.6'
Claude Sonnet 4.6Later report, 30 Jun 2026, in the Claude Sonnet 5 system card: no change stated76.2%AnnouncementSection 8.1 evaluation summary, Table 8.1.A (page 115), row 'BrowseComp', column 'Claude Sonnet 4.6'
Nemotron 3 SuperFirst report, Mar 2026. Setup: internal search harness with SerpAPI; no context management31.28%AnnouncementTable 5 (PDF page 31), row 'BrowseComp with Search', column 'N-3-Super'
Nemotron 3 SuperLater report, 15 Oct 2026, in the Nemotron 3.5 Bolt blog post: different search tool. Setup: Exa web search22.77%AnnouncementTable 7 (PDF page 16), row 'BrowseComp', column 'Nemotron 3 Super 120B-A12B-BF16' (last column)
Claude Mythos PreviewFirst report, Apr 202686.9%Announcementbenchmark table (image), row 'Agentic search, BrowseComp', column 'Mythos Preview'
Claude Mythos PreviewLater report, 9 Jun 2026, in the Claude Fable 5 and Claude Mythos 5 system card: no change stated87.9%AnnouncementSection 8.1 evaluation summary table (page 251), row 'BrowseComp', column 'Mythos Preview'
Claude Opus 4.7First report, Apr 2026. Setup: no thinking, max effort; 10M-token limit, compaction at 200K tokens79.3%Announcementbenchmark table (image), row 'Agentic search, BrowseComp', column 'Opus 4.7'
Claude Opus 4.7Later report, 28 May 2026, in the Claude Opus 4.8 system card: grading changed. Setup: re-run with a new blocklist, compaction at 200K tokens and adaptive thinking79.8%AnnouncementSection 8.1 evaluation summary table (page 194), row 'BrowseComp', column 'Claude Opus 4.7'
Nemotron 3 UltraFirst report, Jun 2026. Setup: internal search harness44.4%Announcementbenchmark table, row 'BrowseComp', column 'Nemotron 3 Ultra BF16'
Nemotron 3 UltraLater report, 15 Oct 2026, in the Nemotron 3.5 Super and Lightning blog post: different harness. Setup: OpenCode harness61.06%AnnouncementTable 10 (PDF page 23), row 'BrowseComp (OpenCode)', column 'N3-Ultra 550B-A55B' (second column)
Nemotron 3.5 LightningFirst report, Aug 202636.97%Announcement'Reasoning Benchmark Evaluations' table, row 'BrowseComp', column 'Nemotron-3.5-Lightning-30B-A3B-BF16'
Nemotron 3.5 LightningLater report, 15 Oct 2026, in the Nemotron 3.5 Super and Lightning blog post: different runs or shots. Setup: average of 2 runs39.50%AnnouncementTable 22 (PDF page 42), row 'BrowseComp', column 'BF16 acc.'

DeepSWE v1.1

ModelScoreSources
Muse Spark 1.1First report, Jul 2026. Setup: xhigh effort; harness based on mini-swe-agent; no internet; averaged over 5 attempts53.3%Announcementbenchmark table (image), row 'DeepSWE 1.1', column 'Muse Spark 1.1'
Muse Spark 1.1Later report, 5 Aug 2026, in the Muse Spark 1.2 and Muse Code post: no change stated. Setup: mini-swe-agent harness; xhigh effort; no internet; averaged over 5 attempts53.0%Announcementbar chart 'DeepSWE 1.1' (image, Coding tab), value label above the bar 'Muse Spark 1.1, mini-swe-agent'
Gemini 3.6 Flash and 3.5 Flash-LiteFirst report, Jul 202649%Announcementchart 'Gemini 3.6 Flash: Outperforms prior generations across agentic benchmarks' (image), panel 'Long-horizon software engineering, DeepSWE v1.1', bar '3.6 Flash' value label
Gemini 3.6 Flash and 3.5 Flash-LiteLater report, 13 Aug 2026, in the Gemini 3.7 Flash post: no change stated48.6%Announcementbenchmark table (image), row 'DeepSWE v1.1', column 'Gemini 3.6 Flash'
Muse Spark 1.2 and Muse CodeFirst report, Aug 2026. Setup: Muse Code harness; xhigh effort; no internet; averaged over 5 attempts59.3%Announcementbar chart 'DeepSWE 1.1' (image, Coding tab), value label above the bar 'Muse Spark 1.2, Muse Code'
Muse Spark 1.2 and Muse CodeLater report, 2 Sep 2026, in the Muse Spark 1.3 post: different harness. Setup: Datacurve leaderboard; xhigh effort; mini-swe-agent harness55.0%AnnouncementMuse Spark 1.3 scorecard (image), row 'DeepSWE v1.1', column 'Muse Spark 1.2 (xhigh)'

FrontierCode 1.1, main set

ModelScoreSources
Claude Opus 5First report, Jul 2026. Setup: run and scored by Cognition; medium effort, the best of five effort levels; mean of 5 runs53.4%Announcementbenchmark table (image), row 'Agentic coding, FrontierCode v1.1, Main', column 'Opus 5'
Claude Opus 5Later report, 22 Sep 2026, in the Claude Opus 5.5 post: different harness. Setup: run by Cognition in Claude Code48.0%Announcementbenchmark table (HTML), row 'Agentic coding, FrontierCode v1.1 (Main)', column 'Opus 5'
Claude Fable 5.1 and Mythos 5.1First report, Sep 2026. Setup: medium effort50.9%AnnouncementSection 8.4 'FrontierCode', paragraph after the task description, sentence on FrontierCode 1.1 (Main)
Claude Fable 5.1 and Mythos 5.1Later report, 22 Sep 2026, in the Claude Opus 5.5 post: different harness. Setup: run by Cognition in Claude Code50.3%Announcementbenchmark table (HTML), row 'Agentic coding, FrontierCode v1.1 (Main)', column 'Fable 5.1'
Claude Fable 5.1 and Mythos 5.1First report, Sep 2026. Setup: medium effort50.9%AnnouncementSection 8.4 'FrontierCode', paragraph after the task description, sentence on FrontierCode 1.1 (Main)
Claude Fable 5.1 and Mythos 5.1Later report, 22 Sep 2026, in the Claude Opus 5.5 system card: different effort. Setup: best effort level; run by Cognition in Claude Code52.8%AnnouncementSection 8.4 'FrontierCode', paragraph ranking Opus 5.5, the Claude Fable 5.1 figure in the FrontierCode (Main) sentence

GPQA Diamond

ModelScoreSources
GPT-4oFirst report, May 202453.6%Announcementtext evaluation bar chart (SVG), panel 'GPQA (%)', bar 'GPT-4o'
GPT-4oLater report, 12 Sep 2024, in the Learning to reason with LLMs post: no change stated. Setup: pass@150.6%Announcementeval table (GPT-4o, o1-preview, o1), row 'GPQA Diamond', sub-row 'pass@1', column 'GPT-4o'
Llama 3.1 405BFirst report, Jul 2024. Setup: 0-shot chain of thought51.1%Announcementbenchmark table (image), row 'GPQA (0-shot, CoT)', column 'Llama 3.1 405B'
Llama 3.1 405BLater report, 5 Apr 2025, in the Llama 4 Scout and Maverick post: different runs or shots. Setup: 0-shot; averaged over multiple generations49.0%Announcement'Llama 4 Scout instruction-tuned benchmarks' table (image), row 'GPQA Diamond', column 'Llama 3.1 405B'
o1 for developersFirst report, Dec 202475.7%Announcementeval table (Category / Eval / o1-2024-12-17 / o1-preview), row 'General: GPQA diamond', column 'o1-2024-12-17'
o1 for developersLater report, 16 Apr 2025, in the o3 and o4-mini post: different effort. Setup: high reasoning effort; o1 version not stated78.0%Announcement'What's changed' section, bar chart 'GPQA Diamond / PhD-Level Science Questions', 1st bar 'o1'
OpenAI o3-miniFirst report, Jan 2025. Setup: high reasoning effort79.7%Announcementimage GPQA.png (bar chart, accuracy by OpenAI model), value label over the 'o3-mini (high)' bar
OpenAI o3-miniLater report, 14 Apr 2025, in the GPT-4.1 post: grading changed. Setup: high reasoning effort; answers parsed by a model, not regex77.2%Announcementacademic eval table (columns GPT-4.1 ... GPT-4.5), row 'GPQA Diamond1', column 'OpenAI o3-mini(high)'
OpenAI o3-miniFirst report, Jan 2025. Setup: high reasoning effort79.7%Announcementimage GPQA.png (bar chart, accuracy by OpenAI model), value label over the 'o3-mini (high)' bar
OpenAI o3-miniLater report, 16 Apr 2025, in the o3 and o4-mini post: no change stated. Setup: effort not stated; no tools77.0%Announcement'What's changed' section, bar chart 'GPQA Diamond / PhD-Level Science Questions', 2nd bar 'o3-mini' (page screenshot (job 20879 capture), part 02)
GPT-4.5First report, Feb 202571.4%Announcementeval table, row 'GPQA (science)', column 'GPT‑4.5'
GPT-4.5Later report, 14 Apr 2025, in the GPT-4.1 post: grading changed. Setup: answers parsed by a model, not regex69.5%Announcementacademic eval table (columns GPT-4.1 ... GPT-4.5), row 'GPQA Diamond1', column 'GPT-4.5' (last column)
Llama NemotronFirst report, Jul 202566.7%AnnouncementFigure 2 bar chart (image), value label of the 'Llama Nemotron Super 49B v1' bar (lightest, rightmost in group) in group 'GPQA Diamond'
Llama NemotronLater report, 5 Mar 2026, in the NVFP4 quantization-aware distillation report: different eval tools. Setup: BF16 baseline of an NVFP4 quantization study; eval tools differ from the model card; average of 20 runs66.5%AnnouncementTable 2 (SFT-heavy models, PDF page 5), model 'Llama Nemotron Super V1', row 'BF16', column 'GPQA-D'
Gemini 2.5 ProFirst report, Mar 2025. Setup: pass@184.0%Announcementbenchmark table (image), row 'Science, GPQA diamond', 'single attempt (pass@1)', column 'Gemini 2.5 Pro Experimental (03-25)'
Gemini 2.5 ProLater report, 18 Nov 2025, in the Start building with Gemini 3 post: no change stated86.4%Announcementbenchmark table (image), row 'GPQA Diamond', 'No tools', column 'Gemini 2.5 Pro'
Mistral Medium 3First report, May 2025. Setup: 5-shot; chain of thought57.1%Announcementbenchmark table image ('Table Medium 6'), section KNOWLEDGE, row 'GPQA Diamond 5-shot CoT', column 'Mistral Medium 3'
Mistral Medium 3Later report, 10 Jun 2025, in the Magistral post: no change stated. Setup: shots and chain of thought not stated59.6%Announcementlater report: Magistral post, bar chart image ('Plot Magistral'), group 'GPQA Diamond', first bar (legend 'Mistral-Medium 3'), value label
GLM-4.5 and GLM-4.5-AirFirst report, Jul 2025. Setup: thinking mode; average over 8 samples79.1%Announcement'Reasoning' section table, row 'GPQA', column 'GLM-4.5'
GLM-4.5 and GLM-4.5-AirLater report, 30 Sep 2025, in the GLM-4.6 post: no change stated. Setup: 128K context79.9%AnnouncementGLM-4.6 post, benchmark chart image (eight panels), panel 'GPQA', second bar 'GLM-4.5', value label
Claude Opus 4.1First report, Aug 2025. Setup: extended thinking up to 64K tokens80.9%Announcementbenchmark table (image), row 'Graduate-level reasoning, GPQA Diamond', column 'Claude Opus 4.1'
Claude Opus 4.1Later report, 24 Nov 2025, in the Claude Opus 4.5 post: no change stated81.0%Announcementbenchmark table (image), row 'Graduate-level reasoning, GPQA Diamond', column 'Opus 4.1'
Nemotron 3First report, Dec 2025. Setup: no tools73.04%AnnouncementTable 3 (PDF page 22), row 'GPQA (no tools)', column 'N-3-Nano'
Nemotron 3Later report, 11 Aug 2026, in the Nemotron 3.5 Lightning 30B-A3B model card: no change stated. Setup: later re-run; no tools74.05%Announcement'Reasoning Benchmark Evaluations' table, row 'GPQA Diamond (no tools)', column 'Nemotron 3 Nano*'
GPT-5.4First report, Mar 202692.8%Announcementacademic table, row 'GPQA Diamond', column 'GPT‑5.4'
GPT-5.4Later report, 17 Mar 2026, in the GPT-5.4 mini and nano post: different effort. Setup: xhigh effort93.0%Announcementacademic table (columns GPT-5.4 (xhigh), GPT-5.4 mini (xhigh), GPT-5.4 nano (xhigh), GPT-5 mini), row 'GPQA Diamond', column 'GPT-5.4 (xhigh)'
Nemotron 3 SuperFirst report, Mar 2026. Setup: no tools79.23%AnnouncementTable 5 (PDF page 31), row 'GPQA (no tools)', column 'N-3-Super'
Nemotron 3 SuperLater report, 15 Oct 2026, in the Nemotron 3.5 Bolt blog post: no change stated78.60%AnnouncementTable 7 (PDF page 16), row 'GPQA Diamond', column 'Nemotron 3 Super 120B-A12B-BF16' (last column)
Nemotron 3.5 LightningFirst report, Aug 2026. Setup: no tools75.44%Announcement'Reasoning Benchmark Evaluations' table, row 'GPQA Diamond (no tools)', column 'Nemotron-3.5-Lightning-30B-A3B-BF16'
Nemotron 3.5 LightningLater report, 15 Oct 2026, in the Nemotron 3.5 Super and Lightning blog post: different runs or shots. Setup: no tools; average of 2 runs76.45%AnnouncementTable 22 (PDF page 42), row 'GPQA Diamond (no tools)', column 'BF16 acc.'

HLE, no tools

ModelScoreSources
OpenAI o3-miniFirst report, Feb 2025. Setup: high reasoning effort13.0%AnnouncementHumanity's Last Exam table (Model / Accuracy (%)), row 'OpenAI o3-mini (high)*'
OpenAI o3-miniLater report, 16 Apr 2025, in the o3 and o4-mini post: no change stated. Setup: effort not stated13.40%Announcement'What's changed' section, bar chart 'Humanity's Last Exam', 2nd bar 'o3-mini' (page screenshot (job 20879 capture), part 02)
Gemini 2.5 ProFirst report, Mar 2025. Setup: pass@118.8%Announcementbenchmark table (image), row 'Reasoning & knowledge, Humanity's Last Exam (no tools)', column 'Gemini 2.5 Pro Experimental (03-25)'
Gemini 2.5 ProLater report, 18 Nov 2025, in the Start building with Gemini 3 post: no change stated21.6%Announcementbenchmark table (image), row 'Humanity's Last Exam', 'No tools', column 'Gemini 2.5 Pro'
o3 and o4-miniFirst report, Apr 2025. Setup: high reasoning effort20.32%Announcement'What's changed' section, bar chart 'Humanity's Last Exam', 3rd bar 'o3 (no tools)' (page screenshot (job 20879 capture), part 02)
o3 and o4-miniLater report, 7 Aug 2025, in the GPT-5 post: different task set or version. Setup: rerun on a later version of HLE; full set14.7%Announcement'Evaluations' section, bar chart 'Humanity's Last Exam (Full Set)*', 9th bar 'OpenAI o3 (no tools)' (page screenshot (job 20879 capture), part 05)
o3 and o4-miniFirst report, Apr 2025. Setup: high reasoning effort20.32%Announcement'What's changed' section, bar chart 'Humanity's Last Exam', 3rd bar 'o3 (no tools)' (page screenshot (job 20879 capture), part 02)
o3 and o4-miniLater report, 7 Aug 2025, in the GPT-5 for developers post: no change stated. Setup: high reasoning effort20.2%Announcementintelligence table (columns GPT-5 (high) ... GPT-4.1 nano), row 'HLE[1] (no tools)', column 'OpenAI o3 (high)'
Nemotron 3First report, Dec 202510.2%Announcement'Reasoning Benchmark Evaluations' table, row 'HLE (no tools)', column 'NVIDIA-Nemotron-3-Nano-30B-A3B-BF16'
Nemotron 3Later report, 11 Aug 2026, in the Nemotron 3.5 Lightning 30B-A3B model card: no change stated. Setup: later re-run; text-only subset10.89%Announcement'Reasoning Benchmark Evaluations' table, row 'HLE (text-only, no tools)', column 'Nemotron 3 Nano*'
Claude Sonnet 4.6First report, Feb 202633.2%Announcementbenchmark table (image), row 'Multidisciplinary reasoning, Humanity's Last Exam (HLE)', column 'Sonnet 4.6', 'without tools'
Claude Sonnet 4.6Later report, 30 Jun 2026, in the Claude Sonnet 5 post: grading changed. Setup: re-graded with an updated grader model34.6%Announcementbenchmark table (image), row 'Multidisciplinary reasoning, Humanity's Last Exam', column 'Sonnet 4.6', 'no tools'
Nemotron 3 SuperFirst report, Mar 202618.26%Announcementbenchmark table under the accuracy chart, row 'HLE (no tools)', column 'Nemotron-3-Super'
Nemotron 3 SuperLater report, 15 Oct 2026, in the Nemotron 3.5 Bolt blog post: no change stated20.30%AnnouncementTable 7 (PDF page 16), row 'HLE (no tools)', column 'Nemotron 3 Super 120B-A12B-BF16' (last column)
Nemotron 3 UltraFirst report, Jun 202626.7%Announcementbenchmark table, row 'HLE (no tools)', column 'Nemotron 3 Ultra BF16'
Nemotron 3 UltraLater report, 15 Oct 2026, in the Nemotron 3.5 Super and Lightning blog post: different task set or version. Setup: text-only subset; GPT-5.6 Luna judge29.24%AnnouncementTable 10 (PDF page 23), row 'HLE (no tools, text)', column 'N3-Ultra 550B-A55B' (second column)
Claude Fable 5 and Mythos 5First report, Jul 202656.5%Announcementbenchmark table (image), row 'Multidisciplinary reasoning, Humanity's Last Exam', column 'Fable 5', 'no tools'
Claude Fable 5 and Mythos 5Later report, 1 Sep 2026, in the Claude Fable 5.1 and Mythos 5.1 post: no change stated57.8%Announcementbenchmark table (HTML), row 'Multidisciplinary reasoning, Humanity's Last Exam', column 'Fable 5', 'no tools'
Claude Sonnet 5First report, Jun 202643.2%Announcementbenchmark table (image), row 'Multidisciplinary reasoning, Humanity's Last Exam', column 'Sonnet 5', 'no tools'
Claude Sonnet 5Later report, 28 Sep 2026, in the Claude Sonnet 5.5 system card: no change stated43.1%AnnouncementSection 8.1 evaluation summary table, row 'Humanity's Last Exam, No tools', column 'Claude Sonnet 5'
Claude Opus 5First report, Jul 202656.3%Announcementbenchmark table (image), row 'Multidisciplinary reasoning, Humanity's Last Exam', column 'Opus 5', 'no tools'
Claude Opus 5Later report, 22 Sep 2026, in the Claude Opus 5.5 system card: no change stated56.6%AnnouncementSection 8.1 evaluation summary table, row 'Humanity's Last Exam, No tools', column 'Claude Opus 5'
Nemotron 3.5 LightningFirst report, Aug 2026. Setup: text-only subset11.72%Announcement'Reasoning Benchmark Evaluations' table, row 'HLE (text-only, no tools)', column 'Nemotron-3.5-Lightning-30B-A3B-BF16'
Nemotron 3.5 LightningLater report, 15 Oct 2026, in the Nemotron 3.5 Super and Lightning blog post: different runs or shots. Setup: average of 2 runs10.80%AnnouncementTable 22 (PDF page 42), row 'HLE (no tools)', column 'BF16 acc.'

HLE, with tools

ModelScoreSources
o3 and o4-miniFirst report, Apr 2025. Setup: high reasoning effort; Python and browsing; known cheating domains blocked; attempts a monitor flags count as wrong24.90%Announcement'What's changed' section, bar chart 'Humanity's Last Exam', 4th bar 'o3 (python + browsing** tools)' (page screenshot (job 20879 capture), part 02)
o3 and o4-miniLater report, 7 Aug 2025, in the GPT-5 post: different task set or version. Setup: rerun on a later version of HLE; full set; Python and browser24.3%Announcement'Evaluations' section, bar chart 'Humanity's Last Exam (Full Set)*', 8th bar 'OpenAI o3 (python + browser)' (page screenshot (job 20879 capture), part 05)
Claude Sonnet 4.5First report, Nov 2025. Setup: with search28.4%AnnouncementFigure 2.16.A 'Humanity's Last Exam (HLE)', panel 'With Search', bar 'Sonnet 4.5' (page 32)
Claude Sonnet 4.5Later report, 5 Feb 2026, in the Claude Opus 4.6 post: no change stated33.6%Announcementbenchmark table (image), row 'Multidisciplinary reasoning, Humanity's Last Exam', column 'Sonnet 4.5', 'with tools'
Claude Opus 4.5First report, Nov 2025. Setup: web search, web fetch and code execution, no reasoning; Claude Sonnet 4.5 as grader43.2%AnnouncementFigure 2.16.A 'Humanity's Last Exam (HLE)', panel 'With Search', bar 'Opus 4.5' (page 32)
Claude Opus 4.5Later report, 5 Feb 2026, in the Claude Opus 4.6 post: no change stated43.4%Announcementbenchmark table (image), row 'Multidisciplinary reasoning, Humanity's Last Exam', column 'Opus 4.5', 'with tools'
Claude Opus 4.6First report, Feb 202653.0%Announcementbenchmark table (image), row 'Multidisciplinary reasoning, Humanity's Last Exam', column 'Opus 4.6', 'with tools'
Claude Opus 4.6Later report, 16 Apr 2026, in the Claude Opus 4.7 post: no change stated53.3%Announcementbenchmark table (image), row 'Multidisciplinary reasoning, Humanity's Last Exam', column 'Opus 4.6', 'with tools'
Claude Sonnet 4.6First report, Feb 2026. Setup: web search, web fetch, code execution, programmatic tool calling; compaction at 50K up to 3M tokens; max effort49.0%Announcementbenchmark table (image), row 'Multidisciplinary reasoning, Humanity's Last Exam (HLE)', column 'Sonnet 4.6', 'with tools'
Claude Sonnet 4.6Later report, 30 Jun 2026, in the Claude Sonnet 5 post: grading changed. Setup: re-graded with an updated grader model46.8%Announcementbenchmark table (image), row 'Multidisciplinary reasoning, Humanity's Last Exam', column 'Sonnet 4.6', 'with tools'
Qwen3.6-PlusFirst report, Apr 2026. Setup: 256K context with context folding50.6%Announcement'Performance' section, 'Language' table, row 'HLE w/ tool', column 'Qwen3.6-Plus'
Qwen3.6-PlusLater report, 20 May 2026, in the Qwen3.7-Max post: no change stated50.2%AnnouncementQwen3.7 post, 'Performance' table, row 'HLE w/ tools', column 'Qwen3.6-Plus'
Claude Fable 5 and Mythos 5First report, Jul 202663.9%Announcementbenchmark table (image), row 'Multidisciplinary reasoning, Humanity's Last Exam', column 'Fable 5', 'with tools'
Claude Fable 5 and Mythos 5Later report, 1 Sep 2026, in the Claude Fable 5.1 and Mythos 5.1 post: no change stated63.8%Announcementbenchmark table (HTML), row 'Multidisciplinary reasoning, Humanity's Last Exam', column 'Fable 5', 'with tools'
Claude Sonnet 5First report, Jun 202657.4%Announcementbenchmark table (image), row 'Multidisciplinary reasoning, Humanity's Last Exam', column 'Sonnet 5', 'with tools'
Claude Sonnet 5Later report, 28 Sep 2026, in the Claude Sonnet 5.5 post: no change stated54.9%Announcementbenchmark table (HTML), row 'Multidisciplinary reasoning, Humanity's Last Exam', column 'Sonnet 5', 'with tools'
Claude Opus 5First report, Jul 202664.7%Announcementbenchmark table (image), row 'Multidisciplinary reasoning, Humanity's Last Exam', column 'Opus 5', 'with tools'
Claude Opus 5Later report, 1 Sep 2026, in the Claude Fable 5.1 and Mythos 5.1 post: no change stated63.6%Announcementbenchmark table (HTML), row 'Multidisciplinary reasoning, Humanity's Last Exam', column 'Opus 5', 'with tools'
Claude Fable 5.1 and Mythos 5.1First report, Sep 202665.0%Announcementbenchmark table (HTML), row 'Multidisciplinary reasoning, Humanity's Last Exam', column 'Fable 5.1', 'with tools'
Claude Fable 5.1 and Mythos 5.1Later report, 22 Sep 2026, in the Claude Opus 5.5 post: no change stated65.6%Announcementbenchmark table, row 'Multidisciplinary reasoning, Humanity's Last Exam', column 'Fable 5.1', 'with tools'

LiveCodeBench

ModelScoreSources
Mistral Medium 3First report, May 2025. Setup: LiveCodeBench v6; 0-shot30.3%Announcementbenchmark table image ('Table Medium 6'), section CODING, row 'LiveCodeBench (v6) 0-shot', column 'Mistral Medium 3'
Mistral Medium 3Later report, 10 Jun 2025, in the Magistral post: different task set or version. Setup: LiveCodeBench v529.1%Announcementlater report: Magistral post, bar chart image ('Plot Magistral'), group 'LiveCodeBench (v5)', first bar (legend 'Mistral-Medium 3'), value label
GLM-4.5 and GLM-4.5-AirFirst report, Jul 2025. Setup: thinking mode; problems from Jul 2024 to Jan 202572.9%Announcement'Reasoning' section table, row 'LiveCodeBench 2407-2501', column 'GLM-4.5'
GLM-4.5 and GLM-4.5-AirLater report, 30 Sep 2025, in the GLM-4.6 post: different task set or version. Setup: LiveCodeBench v6; 128K context63.3%AnnouncementGLM-4.6 post, benchmark chart image (eight panels), panel 'LiveCodeBench v6', second bar 'GLM-4.5', value label
Nemotron 3 UltraFirst report, Jun 2026. Setup: LiveCodeBench v689.0%AnnouncementTable 10 (PDF page 38), row 'LiveCodeBench (v6)', column 'N-3-Ultra'
Nemotron 3 UltraLater report, 15 Oct 2026, in the Nemotron 3.5 Super and Lightning blog post: no change stated. Setup: LiveCodeBench v6, problems from Aug 2024 to May 2025; average of 3 samples89.21%AnnouncementTable 10 (PDF page 23), row 'LiveCodeBench v6', column 'N3-Ultra 550B-A55B' (second column)

MCP Atlas

ModelScoreSources
GLM-5First report, Feb 2026. Setup: 500-task public set; thinking mode; 10-minute timeout per task; Gemini 3 Pro judge67.8%Announcementbenchmark table, row 'MCP-Atlas Public Set', column 'GLM-5 (Thinking)'
GLM-5Later report, 7 Apr 2026, in the GLM-5.1 post: no change stated. Setup: 500-task public set; Gemini 3 Pro judge69.2%Announcementbenchmark table near the end, row 'MCP-Atlas Public Set', column 'GLM-5'

MMLU-Pro

ModelScoreSources
Llama 3.1 405BFirst report, Jul 2024. Setup: 5-shot chain of thought73.3%Announcementbenchmark table (image), row 'MMLU PRO (5-shot, CoT)', column 'Llama 3.1 405B'
Llama 3.1 405BLater report, 5 Apr 2025, in the Llama 4 Scout and Maverick post: different runs or shots. Setup: 0-shot73.4%Announcement'Llama 4 Scout instruction-tuned benchmarks' table (image), row 'MMLU Pro', column 'Llama 3.1 405B'

OSWorld 2.0, offline set, partial score

ModelScoreSources
GPT-5.6First report, Jul 2026. Setup: set, scoring and task release not stated62.6%Announcementcomputer-use/agentic eval table (HTML), row 'OSWorld 2.0', column 'GPT‑5.6 Sol'
GPT-5.6Later report, 3 Sep 2026, in the GPT-6 Astra post: different task set or version. Setup: task release v2026.08.0865.7%Announcement'Computer Use' table, row 'OSWorld 2.0 (v2026.08.08, offline set, partial score)', column 'GPT-5.6 Sol2'
Claude Opus 5First report, Jul 2026. Setup: task release not stated70.6%Announcementbenchmark table image, row 'Computer use / OSWorld 2.0', column 'Opus 5'
Claude Opus 5Later report, 1 Sep 2026, in the Claude Fable 5.1 and Mythos 5.1 post: different task set or version. Setup: partial score; August 2026 task release, not comparable to earlier OSWorld 2.0 scores75.4%Announcementbenchmark table (HTML), row 'Computer use / OSWorld 2.0 [2]', column 'Opus 5', the value tagged 'partial'

SWE-bench Pro

ModelScoreSources
Muse SparkFirst report, Apr 2026. Setup: thinking mode; public set; bash and file-edit tools; averaged over 4 attempts52.4%Announcementmain benchmark table (image), row 'SWE-Bench Pro', column 'Muse Spark Thinking'
Muse SparkLater report, 9 Jul 2026, in the Muse Spark 1.1 post: no change stated55.0%AnnouncementMuse Spark 1.1 benchmark table (image), row 'SWE-Bench Pro', column 'Muse Spark'
GPT-5.5First report, Apr 202658.6%Announcementcoding eval table, row 'SWE-Bench Pro (Public) *', column 'GPT‑5.5'
GPT-5.5Later report, 9 Jul 2026, in the GPT-5.6 post: no change stated59.4%Announcementcoding table (columns GPT-5.6 Sol, Sol Ultra, Terra, Luna, GPT-5.5, ...), row 'SWE-Bench Pro', column 'GPT-5.5'
Qwen3.7-PlusFirst report, Jun 2026. Setup: internal harness with bash and file-edit tools; 200K context57.6%Announcementlanguage benchmark table (first table), Coding Agent block, row 'SWE-Pro', column 'Qwen3.7-Plus'Run on the lab's own corrected version of the public set
Qwen3.7-PlusLater report, 26 Aug 2026, in the Qwen3.8-Flash-Next post: different harness. Setup: Claude Code harness; 256K context55.8%Announcementlater report: Qwen3.8-Flash-Next post, Language table (first table), row 'Agentic coding / SWE-bench Pro', column 'Qwen3.7-Plus'Run on the lab's own corrected version of the public set

SWE-bench Verified

ModelScoreSources
OpenAI o1First report, Sep 2024. Setup: Agentless scaffold; 5 tries to produce a patch; 3-trial average; 477 of the 500 tasks41.4%Announcementsection 4.5.2 'SWE-bench Verified', paragraph under the 'SWE-Bench Verified (with 5-try Agentless Scaffold)' chart, sentence on the final o1-preview model
OpenAI o1Later report, 17 Dec 2024, in the o1 for developers post: no change stated41.3%Announcementeval table (Category / Eval / o1-2024-12-17 / o1-preview), row 'Coding: SWE-bench Verified', column 'o1-preview'
GPT-5First report, Aug 202574.9%Announcementtext, opening summary of results
GPT-5Later report, 13 Nov 2025, in the GPT-5.1 in the API post: different task set or version. Setup: high reasoning effort; all 500 problems72.8%Announcementevaluation table, row 'SWE-bench Verified (all 500 problems)', column 'GPT-5 (high)'
MiniMax M2First report, Oct 202569.4%Announcementfirst benchmark chart image (wide, eight panels), panel 'SWE-bench Verified', first bar 'MiniMax-M2' (red), value label
MiniMax M2Later report, 23 Dec 2025, in the MiniMax M2.1 post: different harness. Setup: Droid (Factory AI) harness68.1%Announcement'Benchmarks' section, second (agent frameworks and specialized benchmarks) table image, row 'SWE-bench Verified (Droid)', column 'MiniMax-M2'
Qwen3.5First report, Feb 202676.4%Announcement'Performance' section, 'Language' table, row 'SWE-bench Verified', column 'Qwen3.5-397B-A17B'
Qwen3.5Later report, 2 Apr 2026, in the Qwen3.6-Plus post: no change stated. Setup: internal harness with bash and file-edit tools; 200K context76.2%AnnouncementQwen3.6-Plus post, 'Performance' section, 'Language' table, row 'SWE-bench Verified', column 'Qwen3.5-397B-A17B'
Nemotron 3 UltraFirst report, Jun 202671.9%Announcementbenchmark table, row 'SWE-Bench Verified', column 'Nemotron 3 Ultra BF16'
Nemotron 3 UltraLater report, 15 Oct 2026, in the Nemotron 3.5 Super and Lightning blog post: different harness. Setup: OpenCode harness; web search and fetch disabled; average of 3 runs73.93%AnnouncementTable 10 (PDF page 23), row 'SWE-bench Verified', column 'N3-Ultra 550B-A55B' (second column)

Terminal-Bench 2.0

ModelScoreSources
Claude Sonnet 4.5First report, Nov 202550.0%Announcementbenchmark table (image), row 'Agentic terminal coding, Terminal-bench 2.0', column 'Sonnet 4.5'
Claude Sonnet 4.5Later report, 5 Feb 2026, in the Claude Opus 4.6 system card: no change stated51.0%AnnouncementTable 2.3.A (PDF page 18), row 'Terminal-Bench 2.0', column 'Claude Sonnet 4.5'
Gemini 3 ProFirst report, Nov 2025. Setup: Terminus 2 harness54.2%Announcementbenchmark table (image), row 'Terminal-Bench 2.0', 'Terminus-2 agent', column 'Gemini 3 Pro'
Gemini 3 ProLater report, 19 Feb 2026, in the Gemini 3.1 Pro post: no change stated. Setup: Terminus 2 harness; Thinking (High)56.9%Announcementbenchmark table (image), row 'Terminal-Bench 2.0', 'Terminus-2 harness', column 'Gemini 3 Pro, Thinking (High)'
Claude Opus 4.5First report, Nov 2025. Setup: 128K thinking budget, high effort; averaged over 5 trials59.3%Announcementbenchmark table (image), row 'Agentic terminal coding, Terminal-bench 2.0', column 'Opus 4.5'
Claude Opus 4.5Later report, 5 Feb 2026, in the Claude Opus 4.6 system card: no change stated59.8%AnnouncementTable 2.3.A (PDF page 18), row 'Terminal-Bench 2.0', column 'Claude Opus 4.5'
GLM-4.7First report, Dec 2025. Setup: Terminus harness41.0%Announcementbenchmark table, row 'Terminal Bench 2.0', column 'GLM-4.7'
GLM-4.7Later report, 12 Feb 2026, in the GLM-5 post: different harness. Setup: Claude Code harness32.8%Announcementbenchmark table, row 'Terminal-Bench 2.0 Claude Code', column 'GLM-4.7 (Thinking)'

Terminal-Bench 2.1

ModelScoreSources
Claude Opus 4.8First report, May 2026. Setup: Terminus 2 harness74.6%Announcementbenchmark table image, row 'Agentic terminal coding / Terminal-Bench 2.1', column 'Opus 4.8'
Claude Opus 4.8Later report, 30 Jun 2026, in the Claude Sonnet 5 post: no change stated82.7%Announcementbenchmark table (image), row 'Agentic coding, Terminal-Bench 2.1', column 'Opus 4.8 (For reference)'
Nemotron 3 UltraFirst report, Jun 202656.4%Announcementbenchmark table, row 'Terminal Bench 2.1', column 'Nemotron 3 Ultra BF16'
Nemotron 3 UltraLater report, 15 Oct 2026, in the Nemotron 3.5 Super and Lightning blog post: no change stated. Setup: Terminus 2 harness; 3h timeout; average of 8 runs54.78%AnnouncementTable 10 (PDF page 23), row 'Terminal-Bench 2.1', column 'N3-Ultra 550B-A55B' (second column)
Muse Spark 1.1First report, Jul 2026. Setup: xhigh effort; bash-only harness; averaged over 5 attempts80.0%Announcementbenchmark table (image), row 'Terminal-Bench 2.1', column 'Muse Spark 1.1'
Muse Spark 1.1Later report, 5 Aug 2026, in the Muse Spark 1.2 and Muse Code post: different harness. Setup: mini-swe-agent harness; xhigh effort; averaged over 5 attempts76.2%Announcementbar chart 'Terminal-Bench 2.1' (image, Coding tab), value label above the bar 'Muse Spark 1.1, mini-swe-agent'

TB-Science 0.1

ModelScoreSources
GPT-6 AstraFirst report, Sep 202664.6%AnnouncementAcademic table, row 'Terminal-Bench Science 0.1', column 'GPT‑6 Astra'
GPT-6 AstraLater report, 29 Sep 2026, in the GPT-6.1 Sol post: different effort. Setup: maximum effort68.1%Announcement'Scientific research' section, paragraph after the Terminal-Bench Science 0.1 cost sentence: 'GPT-6 Astra still achieves the highest score among the models tested at ...'