Frontier Log

AI Labs Report One Benchmark Four Ways

From May 10 to Nov 10, 2026, labs reported Humanity's Last Exam, SWE-bench Pro and OSWorld 2.1 in four setups each, from task sets, scoring rules and tool use.

Six trees, one per benchmark, each splitting into the setups labs reported from May 10 to Nov 10, 2026, with the logos of the labs reporting each. Humanity's Last Exam, 4 setups: no tools or with tools, each on the full set or the text-only subset. OSWorld 2.1, 4 setups: full set or offline set, each with a partial score or a strict pass rate. SWE-bench Pro, 4 setups: the public set, Alibaba's corrected set, Meituan's corrected set and Poolside's patched set. AutomationBench 1.0.6, 3 setups: the private set run by Zapier, the private set run by Artificial Analysis, and the public set. Agents' Last Exam v1.1, 2 setups: pass rate and score. MCP Atlas, 2 setups: the full set and the public set.
From May 10 to Nov 10, 2026, labs reported Humanity's Last Exam, SWE-bench Pro and OSWorld 2.1 in four setups each, AutomationBench 1.0.6 in three, and Agents' Last Exam v1.1 and MCP Atlas in two each. Each logo marks a lab whose document gives a figure in that setup. Download

When labs report the same version of a benchmark, they often report it in different setups. In launch posts, model cards and technical reports from May 10 to Nov 10, 2026, 10 labs reported Humanity's Last Exam in four setups. They ran it with tools or without, and ran each on the full set or on a text-only subset. Labs also reported SWE-bench Pro and OSWorld 2.1 in four setups each, AutomationBench 1.0.6 in three, and Agents' Last Exam v1.1 and MCP Atlas in two each.

Four setups on Humanity's Last Exam

Without tools, Anthropic, Google and Moonshot AI reported the full set. NVIDIA's report gives a full-set figure without saying whether it used tools, so the count leaves it out. Meta, DeepSeek, MiniMax, Z.ai, NVIDIA and Tencent reported the text-only subset.

With tools, Anthropic, Meta, Moonshot AI, Z.ai and Tencent reported the full set. Z.ai, NVIDIA, Meituan and Tencent reported the text-only subset.

Three labs ran their own SWE-bench Pro sets

On SWE-bench Pro, Google, Meta, NVIDIA and Thinking Machines reported the public set of 731 tasks. Alibaba, Meituan and Poolside each reported a set it had corrected or patched itself.

Task sets and scoring on agent tests

OSWorld 2.1 runs on its full set or on an offline set, and each can be scored with a partial score or a strict pass rate. Anthropic reported all four setups, the full set in its Claude Sonnet 5.5 system card and the offline set in its Claude Fable 5.2 system card. Meta reported both scores on the full set, and Tencent reported the partial score. Google reported only the partial score on the offline set. OpenAI's GPT-6.1 Terra post charts the offline set without naming the scoring, so the count leaves it out. An earlier story set out these OSWorld setups model by model.

On AutomationBench 1.0.6, OpenAI, Anthropic and Google printed Zapier's own runs on its private set. That set has 657 tasks, Anthropic's Claude Fable 5.2 system card says. Meta, Mistral, Moonshot AI and Thinking Machines printed Artificial Analysis's runs, on what its methodology calls a private 657-task split of release 1.0.6. Artificial Analysis gives each task "the percentage of objectives the model completed", unless the model breaks a guardrail, its methodology says. DeepSeek, MiniMax, Meituan and Tencent reported the public set. A separate story covers who runs AutomationBench.

On Agents' Last Exam v1.1, Tencent gave a pass rate and Google gave a score. Tencent's post doesn't name the task set. Google's evaluation document says its figures are "self-computed on the 122-task v1.1 benchmark".

Two sizes of MCP Atlas

On MCP Atlas, Google, Meta and Thinking Machines reported the full set of 1,000 tasks. Eight labs reported the public set of 500 tasks, among them Alibaba, MiniMax and Tencent. Meta and Thinking Machines printed figures on both.

How the setups were counted

The count takes every result the site records from the labs' own launch posts, model cards and technical reports dated May 10 to Nov 10, 2026, on the latest version of each benchmark. Each result counts once, for the lab that published the document the site records as its source. That document may give figures for other labs' models. A setup is the task set and the scoring rule or measure that the document, the methodology it cites or another of the lab's documents names. Results whose documents don't name the set, the scoring or the measure are left out. Earlier versions, such as OSWorld 2.0 and earlier AutomationBench releases, are left out too. Other settings change a score as well, such as the agent harness and the effort setting, and the image leaves them out. An earlier story measured how much the harness version moves a score.

Humanity's Last Exam, no tools, full set

ModelScoreSources
Gemini 3.5 FlashGemini 3.5 Flash, reported by Google40.2%Announcementbenchmark table (image), row 'Reasoning, Humanity's Last Exam', column 'Gemini 3.5 Flash'
Kimi K3.1 ProKimi K3.1 Pro, reported by Moonshot AI51.1%Announcement'Benchmarks' table (rendered by the page's script), section 'Reasoning & Knowledge & Multimodal', row 'HLE-Full', column 'Kimi K3.1 Pro (max)'
Claude Fable 5.2Claude Fable 5.2, reported by Anthropic73.2%Claude Fable 5.2 and Claude Mythos 5.2 System CardSection 8.1 Evaluation summary, Table 8.1.A, row 'Humanity's Last Exam, No tools', column 'Fable 5.2'

Humanity's Last Exam, no tools, text-only subset

ModelScoreSources
Nemotron 3 Ultra 550B-A55BNemotron 3 Ultra 550B-A55B, reported by NVIDIA29.24%Nemotron 3.5 Super and Lightning: Open, Efficient Mixture-of-Experts Hybrid Mamba-Transformer Models for Agentic AITable 10 (PDF page 23), row 'HLE (no tools, text)', column 'N3-Ultra 550B-A55B' (second column)
GLM-5.2GLM-5.2, reported by Z.ai40.5%Announcement'Full Benchmark Table', row 'HLE', column 'GLM-5.2'
Muse Glimmer 30BMuse Glimmer 30B, reported by Meta22.0%Announcementbenchmark table (image), group 'General Capabilities and Reasoning', row 'Humanity's Last Exam, Text · No Tools', column 'Muse Glimmer-30B, High Reasoning'
Hy4 previewHy4 preview, reported by Tencent43.4%Announcement'Appendix: Benchmark Results' table image, row 'HLE (no tools, text-only)', column 'Hy4 preview'
DeepSeek-V4.1-Flash-1017DeepSeek-V4.1-Flash-1017, reported by DeepSeek44.0%Announcementbenchmark table image (first image), row 'HLE (w/o tools)', column 'DeepSeek V4.1-Flash-1017', second of the two values (asterisk)
MiniMax M3.1 FlashMiniMax M3.1 Flash, reported by MiniMax43.4%Announcementbenchmark table image (first image), row 'HLE (w/o tools)', column 'MiniMax M3.1 Flash'

Humanity's Last Exam, with tools, full set

ModelScoreSources
Muse Spark 1.1Muse Spark 1.1, reported by Meta62.1%Announcementbenchmark table (image), row 'Humanity's Last Exam (w/ tools)', column 'Muse Spark 1.1'
GLM-5.3-FlashGLM-5.3-Flash, reported by Z.ai55.3%Announcementmain benchmark table (second table), AGENTIC block, row 'HLE w/ Tools', column 'GLM-5.3-Flash'
Kimi K3.1 ProKimi K3.1 Pro, reported by Moonshot AI70.0%Announcement'Benchmarks' table (rendered by the page's script), section 'Agentic', row 'HLE-Full w/ tools', column 'Kimi K3.1 Pro (max)'
Hy4Hy4, reported by Tencent59.1%Announcement'Appendix: benchmark results' table image, row 'HLE-Full (with tools)', column 'Hy4'
Claude Fable 5.2Claude Fable 5.2, reported by Anthropic74.2%Announcementbenchmark table (HTML), row 'Multidisciplinary reasoning, Humanity's Last Exam (with tools)', column 'Fable 5.2'

Humanity's Last Exam, with tools, text-only subset

ModelScoreSources
GLM-5.2GLM-5.2, reported by Z.ai54.7%Announcement'Full Benchmark Table', row 'HLE w/ Tools', column 'GLM-5.2'
Hy4 previewHy4 preview, reported by Tencent55.4%Announcement'Appendix: Benchmark Results' table image, row 'HLE (with tools, text-only)', column 'Hy4 preview'
Nemotron 3.5 Super VL 120B-A12BNemotron 3.5 Super VL 120B-A12B, reported by NVIDIA46.01%Nemotron 3.5 Super and Lightning: Open, Efficient Mixture-of-Experts Hybrid Mamba-Transformer Models for Agentic AITable 10 (PDF page 23), row 'HLE (with tools, text)', column 'N3.5-Super 120B-A12B' (first column)
LongCat-2.5LongCat-2.5, reported by Meituan63.1%Announcement'Evaluations' section table at the end of the post (rendered by the page's script; load https://longcat.ai/blog/longcat-2.5 in a browser and scroll to the end), row 'HLE w/tools' under 'General Agent', column 'LongCat-2.5'

OSWorld 2.1, full set, partial score

ModelScoreSources
Claude Sonnet 5.5Claude Sonnet 5.5, reported by Anthropic80.1%Announcementbenchmark table (HTML), row 'Computer use / OSWorld 2.1', column 'Sonnet 5.5', the value tagged 'partial'
Hy4Hy4, reported by Tencent78.9%Announcement'Appendix: benchmark results' table image, row 'OSWorld 2.1 (Full, partial score)', column 'Hy4'
Muse EmberMuse Ember, reported by Meta69.5%Announcement'Muse Ember evaluation results' table (HTML), row 'Multimodal / OSWorld 2.1', column 'Muse Ember (max)', the value marked '(partial)'

OSWorld 2.1, full set, strict pass rate

ModelScoreSources
Claude Sonnet 5.5Claude Sonnet 5.5, reported by Anthropic43.5%Claude Sonnet 5.5 System Cardsection 8.13.3 OSWorld 2.1, text (PDF page 130), the Claude Sonnet 5.5 strict pass rate
Muse EmberMuse Ember, reported by Meta33.8%Announcement'Muse Ember evaluation results' table (HTML), row 'Multimodal / OSWorld 2.1', column 'Muse Ember (max)', the value marked '(binary)'

OSWorld 2.1, offline set, partial score

ModelScoreSources
Claude Fable 5.1Claude Fable 5.1, reported by Anthropic83.5%Introducing Claude Fable 5.2 and Claude Mythos 5.2benchmark table (HTML), row 'Computer use, OSWorld 2.1 (offline set, partial)', column 'Fable 5.1'
Gemini 4 ArgonGemini 4 Argon, reported by Google74.7%Gemini 4 Carbon: our next frontier modellater report: Gemini 4 Carbon post, benchmark table (image), row 'OSWorld-2.1', 'Offline subset', column 'Gemini 4 Argon'

OSWorld 2.1, offline set, strict pass rate

ModelScoreSources
Claude Fable 5.1Claude Fable 5.1, reported by Anthropic47.6%Claude Fable 5.2 and Claude Mythos 5.2 System CardSection 8.13.3 'OSWorld 2.1 offline set', paragraph after the setup, the sentence on models re-evaluated under the same conditions, Claude Fable 5.1's strict pass rate

SWE-bench Pro, public set (731 tasks)

ModelScoreSources
InklingInkling, reported by Thinking Machines54.3%Announcementsection 'Benchmarking Inkling', main benchmark table, row 'SWEBench Pro' with subtitle 'Public' (Agentic (coding)), column 'Inkling effort=0.99' (the Inkling vs Inkling-Small table later prints the same value)
Gemini 3.5 Flash-LiteGemini 3.5 Flash-Lite, reported by Google54.2%Announcementchart 'Gemini 3.5 Flash-Lite, Compared to 3 Flash' (image), panel 'Agentic Software Engineering Tasks, SWE-Bench Pro (Public)', bar '3.5 Flash-Lite' value label
Muse Glimmer 30BMuse Glimmer 30B, reported by Meta51.2%Muse Glimmer: an open-weight model built for local agentsbenchmark table (image), row 'SWE-Bench Pro', column 'Muse Glimmer-30B, High Reasoning'
Nemotron 3.5 Super VL 120B-A12BNemotron 3.5 Super VL 120B-A12B, reported by NVIDIA49.45%nvidia/NVIDIA-Nemotron-3.5-Super-VL-120B-A12B-NVFP4 model card (launch-day revision)'Agentic Coding Benchmarks' image, 'SWE-Bench Pro' panel, value label of the 'Claude Code' bar

SWE-bench Pro, Alibaba's corrected set

ModelScoreSources
Qwen3.8-Omni-FlashQwen3.8-Omni-Flash, reported by Alibaba63.3%Announcementtext benchmark table (Coding and Agent block), row 'SWE-bench Pro', column 'Qwen3.8-Omni-Flash'

SWE-bench Pro, Meituan's corrected set

ModelScoreSources
LongCat-2.0LongCat-2.0, reported by Meituan59.5%Announcement'Evaluations' section table (rendered by the page's script; load it in a browser), row 'SWE-bench Pro' under 'CODE AGENT', column 'LongCat-2.0'

SWE-bench Pro, Poolside's patched set

ModelScoreSources
Laguna XS 2.1Laguna XS 2.1, reported by Poolside47.6%Announcementevaluation chart 'SWE-Bench Pro' (Resolved tasks on SWE-Bench Pro), bar 'Laguna XS 2.1 (33B-A3B)', value label (animates up from 0.0 when scrolled into view; also in the bar's aria-label)

AutomationBench 1.0.6, private set, Zapier's runs

ModelScoreSources
GPT-6.1 SolGPT-6.1 Sol, reported by OpenAI36.1%AnnouncementAutomationBench chart under "Professional work" (score vs cost per task; no value labels): point 'GPT-6.1 Sol', effort 'Max', read from the point's aria-label (load the page in a browser and hover or inspect the SVG point)
Gemini 4 CarbonGemini 4 Carbon, reported by Google53.9%Announcementbenchmark table (image), row 'AutomationBench', 'Zapier', column 'Gemini 4 Carbon'
Claude Fable 5.2Claude Fable 5.2, reported by Anthropic51.4%Announcementbenchmark table (HTML), row 'Business workflows, AutomationBench', column 'Fable 5.2'

AutomationBench 1.0.6, private set, Artificial Analysis's runs

ModelScoreSources
Mistral Large 4Mistral Large 4, reported by Mistral59.9%Announcement'Agentic Workflows' section, chart tab 'AA - AutomationBench' (image titled 'Artificial Analysis - AutomationBench'), value label on the 'Mistral Large 4 Preview' bar; the section's text gives the same score for AutomationBench, '657 business workflows'
Kimi K3.1 ProKimi K3.1 Pro, reported by Moonshot AI73.3%Announcement'Benchmarks' table (rendered by the page's script), section 'Agentic', row 'Automation Bench-AA', column 'Kimi K3.1 Pro (max)'; runner: the table's notes, 'AutomationBench-AA scores are cited from https://artificialanalysis.ai/'
Inkling 2 SmallInkling 2 Small, reported by Thinking Machines45.6%Inkling 2 Small Model Cardsection '5. Evaluations', evaluations table, Enterprise Operations group, row 'AutomationBench v1.0.6', column 'Inkling 2 Small' (first model column); set and runner: launch post thinkingmachines.ai/news/inkling-2-small/, list of externally reported evals right after its evaluations table, before 'Limitations'
Muse EmberMuse Ember, reported by Meta37.8%Announcement'Muse Ember evaluation results' table (HTML), row 'Agents / AutomationBench-AA', column 'Muse Ember (max)'

AutomationBench 1.0.6, public set

ModelScoreSources
Hy4 previewHy4 preview, reported by Tencent36.3%Introducing Hy4later report: Hy4 post, 'Appendix: benchmark results' table image, row 'AutomationBench v1.0.6', column 'Hy4 preview'
DeepSeek-V4.1-FlashDeepSeek-V4.1-Flash, reported by DeepSeek54.8%Announcementbenchmark table image (second image), row 'Automation-Bench', column 'DeepSeek V4.1-Flash' (the first chart image, v4.1_260910_benchmark.png, prints the same value label)
MiniMax M3.1 FlashMiniMax M3.1 Flash, reported by MiniMax62.8%Announcementbenchmark table image (first image), row 'Automation-Bench v1.0.6', column 'MiniMax M3.1 Flash'
LongCat-2.5LongCat-2.5, reported by Meituan51.8%Announcement'Evaluations' section table at the end of the post (rendered by the page's script; load https://longcat.ai/blog/longcat-2.5 in a browser and scroll to the end), row 'AutomationBench v1.0.6' under 'General Agent', column 'LongCat-2.5'

Agents' Last Exam v1.1, pass rate

ModelScoreSources
Hy4Hy4, reported by Tencent41.0%Announcement'Appendix: benchmark results' table image, row 'Agents' Last Exam v1.1 (Pass Rate)', column 'Hy4'

Agents' Last Exam v1.1, score

ModelScoreSources
Gemini 4 ArgonGemini 4 Argon, reported by Google69.0%Gemini 4 Carbon: our next frontier modellater report: Gemini 4 Carbon post, benchmark table (image), row 'Agents' Last Exam v1.1', 'Score', column 'Gemini 4 Argon'

MCP Atlas, full set (1,000 tasks)

ModelScoreSources
Gemini 3.5 FlashGemini 3.5 Flash, reported by Google83.6%Announcementbenchmark table (image), row 'Agentic, MCP Atlas', column 'Gemini 3.5 Flash'
Muse Spark Open 816B-A42BMuse Spark Open 816B-A42B, reported by Meta89.2%Introducing Muse Spark Open, The Most Capable US Open-Weight Model to Datescorecard (image), row 'MCP-Atlas', column 'Muse Spark Open (xhigh)'
Inkling 2 SmallInkling 2 Small, reported by Thinking Machines85.4%Announcementsection 'Benchmarking Inkling 2 Small', main benchmark table, Search and Tools group, row 'MCP Atlas' (subtitle 'Full / Public'), column 'Inkling 2 Small' (first model column), first value in the cell (before the slash); runner: list of externally reported evals right after the main benchmark table, before 'Limitations'

MCP Atlas, public set (500 tasks)

ModelScoreSources
MiniMax M2.7MiniMax M2.7, reported by MiniMax49.4%MiniMax M3: Frontier Coding, 1M Context, Native Multimodality — All in One ModelMiniMax M3 post, benchmark table image (tall, nine columns), row 'MCP Atlas', column 'MiniMax M2.7' (second column)
Qwen3.7-PlusQwen3.7-Plus, reported by Alibaba73.2%Announcement'Performance' section, language table (HTML), General Agent block, row 'MCP-Atlas', column 'Qwen3.7-Plus'
GLM-5.2GLM-5.2, reported by Z.ai76.8%Announcement'Full Benchmark Table', row 'MCP-Atlas Public Set', column 'GLM-5.2'
Kimi K3Kimi K3, reported by Moonshot AI84.2%Announcement'Full Benchmark Table', row 'MCP Atlas', column 'Kimi K3 (max)'
Muse Glimmer 30BMuse Glimmer 30B, reported by Meta75.5%Muse Glimmer: an open-weight model built for local agentsbenchmark table (image), row 'MCP Atlas', column 'Muse Glimmer-30B, High Reasoning'
Hy4 previewHy4 preview, reported by Tencent83.7%Announcement'Appendix: Benchmark Results' table image, row 'MCP-Atlas (public)', column 'Hy4 preview'
Nemotron 3.5 Super VL 120B-A12BNemotron 3.5 Super VL 120B-A12B, reported by NVIDIA51.20%Nemotron 3.5 Bolt 74B-A9B 24GB: A Hybrid Vision-Language Model for Local AIlater report: Nemotron 3.5 Bolt technical report, Table 7, row 'MCP-Atlas', column 'Nemotron 3.5 Super 120B-A12B-BF16'
Inkling 2 SmallInkling 2 Small, reported by Thinking Machines85.0%Announcementsection 'Benchmarking Inkling 2 Small', main benchmark table, Search and Tools group, row 'MCP Atlas' (subtitle 'Full / Public'), column 'Inkling 2 Small' (first model column), second value in the cell (after the slash); runner: list of externally reported evals right after the main benchmark table, before 'Limitations'

Sources

Models

Related

Tags