Frontier Log

Who reports which benchmark

Who reports which coding benchmark. 25 key launches from seven labs (OpenAI, Anthropic, Google, DeepSeek, Qwen, Moonshot, MiniMax), Jan 2025 to Sep 2026, one row each in date order; a filled dot where the launch post or a card it links reports the benchmark, a ring where it leaves it out. Columns: LiveCodeBench, SWE-bench Verified and Pro, DeepSWE v1.1, Terminal-Bench 2.0, 2.1 and 4.0. Early launches report LiveCodeBench and SWE-bench Verified; the September 2026 launches drop both and report DeepSWE v1.1 and Terminal-Bench 4.0.
Early launches reported LiveCodeBench and SWE-bench Verified. By September 2026 the key launches had dropped both, reporting DeepSWE v1.1 and Terminal-Bench 4.0 instead. Each row is one of 25 key launches from seven labs, January 2025 to September 2026, with a filled dot where the launch post or a card it links reports the benchmark.

Download

Who reports which knowledge benchmark. 25 key launches from seven labs (OpenAI, Anthropic, Google, DeepSeek, Qwen, Moonshot, MiniMax), Jan 2025 to Sep 2026, one row each in date order; a filled dot where the launch post or a card it links reports the benchmark, a ring where it leaves it out. Columns: MMLU-Pro, GPQA Diamond, Humanity's Last Exam. MMLU-Pro is reported mostly by the open labs; GPQA Diamond and Humanity's Last Exam by most launches until Claude Opus 5.5 and Gemini 4 Argon leave GPQA out.
MMLU-Pro is reported mostly by the open labs. GPQA Diamond and Humanity's Last Exam were reported by most launches, until Claude Opus 5.5 and Gemini 4 Argon left GPQA Diamond out.

Download

Who reports which math benchmark. 25 key launches from seven labs (OpenAI, Anthropic, Google, DeepSeek, Qwen, Moonshot, MiniMax), Jan 2025 to Sep 2026, one row each in date order; a filled dot where the launch post or a card it links reports the benchmark, a ring where it leaves it out. Columns: AIME 2024, AIME 2025, HMMT 2025. AIME 2024 gives way to AIME 2025 in 2025; from April 2026 no key launch reports any of the three.
AIME 2024 gave way to AIME 2025 during 2025. Since April 2026, no key launch has reported AIME 2024, AIME 2025 or HMMT 2025.

Download

Who reports which agent benchmark. 25 key launches from seven labs (OpenAI, Anthropic, Google, DeepSeek, Qwen, Moonshot, MiniMax), Jan 2025 to Sep 2026, one row each in date order; a filled dot where the launch post or a card it links reports the benchmark, a ring where it leaves it out. Columns: BrowseComp, τ²-bench, MCP Atlas, AutomationBench. Reporting is patchy until AutomationBench, which every launch since July 2026 reports.
Agent benchmarks were reported patchily until AutomationBench, which every key launch since July 2026 has reported.

Download

Best reported scores: the highest score any lab has reported on four benchmarks, by release date, each line ending at the last key launch that reports it. AIME 2025 reaches 100% (GPT-5.2, Dec 2025) and is last reported in Feb 2026; SWE-bench Verified reaches 95% (Claude Fable 5, Jun 2026) and is last reported that month; GPQA Diamond reaches 96% and is still reported; Humanity's Last Exam, at 64.4%, is still reported. Restricted-access models (Mythos) left out.
The best score any lab has reported on four benchmarks. AIME 2025 reached 100% with GPT-5.2 in December 2025 and was last reported in February 2026; SWE-bench Verified reached 95% with Claude Fable 5 in June 2026 and was last reported that month. GPQA Diamond, at 96%, and Humanity's Last Exam, at 64.4%, are still reported. Restricted-access models such as Mythos are left out.

Download

High scores, different exits: ten benchmarks by best reported score; the dot shows how many of seven labs' latest launches report each (OpenAI, Anthropic, Google, DeepSeek, Qwen, Moonshot, Mistral): a ring for none, half filled for fewer than half, filled for half or more. No longer reported: AIME 2025 (100%, saturated), τ²-bench Telecom (99.3%, saturated, replaced by τ³-bench), SWE-bench Verified (95%, saturated and flawed) and Terminal-Bench 2.0 (82.7%, replaced by versions 2.1 and 4.0). Partly reported: GPQA Diamond (96%, saturated; OpenAI, Qwen and Moonshot) and SWE-bench Pro (89.9%, flawed; Anthropic and Qwen). Still reported: DeepSWE v1.1 (77.9%), Terminal-Bench 4.0 (70.6%), Humanity's Last Exam (64.4%) and AutomationBench (51.3%). Each reason is cited to its source at the bottom of the image.
Why benchmarks drop out of launch posts. AIME 2025, τ²-bench Telecom, SWE-bench Verified and Terminal-Bench 2.0 are no longer reported: saturated, flawed or replaced by a newer version. GPQA Diamond and SWE-bench Pro are reported by only a few labs, while DeepSWE v1.1, Terminal-Bench 4.0, Humanity's Last Exam and AutomationBench are still reported by most. Each reason is cited at the bottom of the image.

Download

Why

BenchmarkBest scoreBest by
AIME 2025Released 12 Feb 2025
GPT-5.2 scored 100% in Dec 2025 [1]
Anthropic's system cards call it saturated [2]		100.0%GPT-5.2
τ²-bench TelecomReleased 9 Jun 2025
Its successor, τ³-bench, came out in Mar 2026 [3]
99.3%Claude Opus 4.6
SWE-bench VerifiedReleased 13 Aug 2024
OpenAI, its creator, stopped reporting it in Feb 2026, citing flawed tests [4]
95%Claude Fable 5 and Mythos 5
Terminal-Bench 2.0Released 7 Nov 2025
Its creators released versions 2.1 (May 2026) and 4.0 (Aug 2026) [5][6]
82.7%GPT-5.5
GPQA DiamondReleased 20 Nov 2023
Anthropic called it saturated in Jun 2026 [7]
96.0%GPT-6 Astra
SWE-bench ProReleased 19 Sep 2025
OpenAI estimated in Jul 2026 that about 30% of its tasks are broken [8]
89.9%Claude Opus 5.5
DeepSWE v1.1Released 14 Jun 202677.9%Gemini 4 Argon
Terminal-Bench 4.0Released 28 Aug 202670.6%Claude Sonnet 5.5
Humanity's Last ExamReleased 23 Jan 202564.4%Claude Opus 5.5
AutomationBenchReleased 20 Apr 202651.3%Gemini 4 Argon

Sources

  1. OpenAI, Introducing GPT-5.2 (Dec 2025)
  2. Anthropic, Claude Opus 4.7 system card (Apr 2026)
  3. Sierra, τ³-bench (Mar 2026)
  4. OpenAI, Why we no longer evaluate SWE-bench Verified (Feb 2026)
  5. Terminal-Bench, Terminal-Bench 2.1 (May 2026)
  6. Terminal-Bench, Terminal-Bench 4.0 (Aug 2026)
  7. Anthropic, Claude Fable 5 and Mythos 5 system card (Jun 2026)
  8. OpenAI, Separating signal from noise in coding evaluations (Jul 2026)

GPQA Diamond

ModelScoreSources
Claude 3 family33.3%Announcementbenchmark table (image), row 'Graduate level reasoning, GPQA, Diamond', column 'Claude 3 Haiku'
Claude 3 family40.4%Announcementbenchmark table (image), row 'Graduate level reasoning, GPQA, Diamond', column 'Claude 3 Sonnet'
Claude 3 family50.4%Announcementbenchmark table (image), row 'Graduate level reasoning, GPQA, Diamond', column 'Claude 3 Opus'
GPT-4o53.6%Announcementtext evaluation bar chart (SVG), panel 'GPQA (%)', bar 'GPT-4o'
Claude 3.5 Sonnet59.4%Announcementbenchmark table (image), row 'Graduate level reasoning, GPQA, Diamond', column 'Claude 3.5 Sonnet' (value marked *)
Gemini 2.0 Flash62.1%Announcementbenchmark table (image), row 'Reasoning, GPQA (diamond)', column 'Gemini 2.0 Flash Experimental'
Gemini 2.0 Flash, Flash-Lite and Pro64.7%Announcementbenchmark table (image), row 'Reasoning, GPQA (diamond)', column 'Gemini 2.0 Pro' (experimental)
Claude 3.7 Sonnet78.2%Announcementbenchmark table (image), row 'Graduate-level reasoning, GPQA Diamond', column 'Claude 3.7 Sonnet, 64K extended thinking', first value
Gemini 2.5 Pro84.0%Announcementbenchmark table (image), row 'Science, GPQA diamond', 'single attempt (pass@1)', column 'Gemini 2.5 Pro Experimental (03-25)'
GPT-5.188.1%Announcementfirst comparison table, column 'GPT‑5.1 Thinking', row 'GPQA Diamond (no tools)'
Gemini 3 Pro91.9%Announcementbenchmark table (image), row 'GPQA Diamond', 'No tools', column 'Gemini 3 Pro'
GPT-5.292.4%Announcementscience and math table, row 'GPQA Diamond (no tools)', column 'GPT-5.2 Thinking'
Gemini 3.1 Pro94.3%Announcementbenchmark table (image), row 'GPQA Diamond', 'No tools', column 'Gemini 3.1 Pro'
GPT-5.694.6%Announcementacademic eval table, column 'GPT‑5.6 Sol', row 'GPQA Diamond'
GPT-6 Astra96.0%Announcement'Academic' table, column 'GPT‑6 Astra', row 'GPQA Diamond'

SWE-bench Verified

ModelScoreSources
GPT-4o30.7%Announcementeval table, row 'SWE-Bench Verified (coding)*', column 'GPT‑4o'
DeepSeek-R149.2%Announcementbar chart image (first image), group 'SWE-bench Verified (Resolved)', first bar (legend 'DeepSeek-R1'), value label
Claude 3.7 Sonnet62.3%Announcementbenchmark table (image), row 'Agentic coding, SWE-bench Verified', column 'Claude 3.7 Sonnet, No extended thinking', first value
o3 and o4-mini69.1%AnnouncementCoding section, bar chart 'SWE-Bench Verified (n=477)', bar labelled 'o3' (3rd bar)
Claude 4 (Opus 4 and Sonnet 4)72.5%Announcement'Software engineering, SWE-bench verified' bar chart (image), bar 'Opus 4', value inside the bar
Claude 4 (Opus 4 and Sonnet 4)72.7%Announcement'Software engineering, SWE-bench verified' bar chart (image), bar 'Sonnet 4', value inside the bar
Claude Opus 4.174.5%Announcementbenchmark table (image), row 'Agentic coding, SWE-bench Verified', column 'Claude Opus 4.1'
GPT-574.9%Announcementtext, opening summary of results
Claude Sonnet 4.577.2%Announcementbenchmark table (image), row 'Agentic coding, SWE-bench Verified', column 'Claude Sonnet 4.5', first value
Claude Opus 4.580.9%Announcementbenchmark table (image), row 'Agentic coding, SWE-bench Verified', column 'Opus 4.5'
Claude Opus 4.787.6%Announcementbenchmark table (image), row 'Agentic coding, SWE-bench Verified', column 'Opus 4.7'
Claude Opus 4.888.6%AnnouncementSection 8.1 evaluation summary table (page 194), row 'SWE-bench Verified', column 'Claude Opus 4.8'
Claude Fable 5 and Mythos 595%AnnouncementSection 8.1 evaluation summary table (page 251), row 'SWE-bench Verified', column 'Fable 5'

AIME 2025

ModelScoreSources
Gemini 2.5 Pro86.7%Announcementbenchmark table image (gemini_benchmarks_cropped GIF), row 'Mathematics AIME 2025', sub-row 'single attempt (pass@1)', column 'Gemini 2.5 Pro Experimental (03-25)'
o3 and o4-mini88.9%Announcement'What's changed' section, bar chart 'AIME 2025 / Competition Math', bar labelled 'o3 (no tools)' (3rd bar)
GPT-594.6%Announcement'Evaluations' section, first paragraph (state of the art across math)
GPT-5.2100.0%Announcementheadline eval table (HTML, first table), row 'AIME 2025 (no tools) / Competition math', column 'GPT-5.2 Thinking'

Humanity's Last Exam, no tools

ModelScoreSources
Gemini 2.5 Pro18.8%Announcementbenchmark table (image), row 'Reasoning & knowledge, Humanity's Last Exam (no tools)', column 'Gemini 2.5 Pro Experimental (03-25)'
Kimi K2 Thinking23.9%Announcement'Full Evaluations' table, row 'Humanity's Last Exam (Text-only)', sub-row 'no tools', column 'K2 Thinking'
GPT-5.125.7%Announcementscience and math eval table, column 'GPT-5.1 Thinking', row 'HLE (no tools)'
Gemini 3 Pro37.5%Announcementbenchmark table (image), row 'Humanity's Last Exam', 'No tools', column 'Gemini 3 Pro'
Claude Opus 4.640.0%Announcementbenchmark table (image), row 'Multidisciplinary reasoning, Humanity's Last Exam', column 'Opus 4.6', 'without tools'
Gemini 3.1 Pro44.4%Announcementbenchmark table (image), row 'Humanity's Last Exam', 'No tools', column 'Gemini 3.1 Pro, Thinking (High)'
Claude Opus 4.746.9%Announcementbenchmark table (image), row 'Multidisciplinary reasoning, Humanity's Last Exam', column 'Opus 4.7', 'no tools'
Claude Opus 4.849.8%Announcementbenchmark table (image), row 'Multidisciplinary reasoning, Humanity's Last Exam', column 'Opus 4.8', 'no tools'
Claude Fable 5 and Mythos 556.5%Announcementbenchmark table (image), row 'Multidisciplinary reasoning, Humanity's Last Exam', column 'Fable 5', 'no tools'
Claude Fable 5.1 and Mythos 5.160.9%Announcementbenchmark table (HTML), row 'Multidisciplinary reasoning, Humanity's Last Exam', column 'Fable 5.1', 'no tools'
Claude Opus 5.564.4%AnnouncementSection 8.1 evaluation summary table, row 'Humanity's Last Exam, No tools', column 'Claude Opus 5.5'

SWE-bench Pro

ModelScoreSources
Claude Opus 5.589.9%AnnouncementSection 8.1 Evaluation summary, Table 8.1.A, row 'SWE-bench Pro', column 'Claude Opus 5.5'The source doesn't say whether this is the public or the commercial set

DeepSWE v1.1

ModelScoreSources
Gemini 4 Argon77.9%Announcementbenchmark table (image), row 'DeepSWE v1.1', column 'Gemini 4 Argon'

Terminal-Bench 2.0

ModelScoreSources
GPT-5.582.7%Announcementheadline eval table (HTML, first table), row 'Terminal-Bench 2.0', column 'GPT‑5.5'

Terminal-Bench 4.0

ModelScoreSources
Claude Sonnet 5.570.6%Announcementbenchmark table, row 'Terminal-Bench 4.0', column 'Sonnet 5.5'

τ²-bench Telecom

ModelScoreSources
Claude Opus 4.699.3%Announcementbenchmark table (image), row 'Agentic tool use, τ2-bench', Telecom line, column 'Opus 4.6'

AutomationBench

ModelScoreSources
Gemini 4 Argon51.3%Announcementbenchmark table (image), row 'AutomationBench', 'Score', column 'Gemini 4 Argon'

Launches

ModelSources
DeepSeek R1Launch postgithub.com
Gemini 2.5 ProLaunch post
Qwen3Launch post
Claude Opus 4Launch post
Kimi K2Launch post
GPT-5Launch post
MiniMax M2Launch posthuggingface.co
Gemini 3 ProLaunch postblog.google
Claude Opus 4.5Launch postwww.anthropic.com
DeepSeek V3.2Launch posthuggingface.co
GPT-5.2Launch post
Kimi K2.5Launch post
MiniMax M2.5Launch post
Qwen3.5Launch post
Gemini 3.1 ProLaunch post
Claude Opus 4.7Launch postwww.anthropic.com
GPT-5.5Launch post
DeepSeek V4Launch post
MiniMax M3Launch post
Kimi K3Launch post
Qwen3.8-MaxLaunch post
GPT-6 AstraLaunch post
DeepSeek V4.1-FlashLaunch post
Claude Opus 5.5Launch postwww.anthropic.com
Gemini 4 ArgonLaunch post

Gemini 4 Argon on the timeline