Frontier Log

Google Takes the Max on OSWorld 2.1

Google reports Gemini 4's OSWorld 2.1 scores as the maximum over three runs. It averages its Terminal-Bench runs, and Anthropic averages OSWorld 2.1.

A table with a row per benchmark and four columns: Average only, Both, Max only and Not stated. OSWorld 2.1: Anthropic under Average only, Google's Gemini 4 Hydrogen under Max only, and OpenAI, Meta, Moonshot AI and Tencent under Not stated. OSWorld 2.0: Anthropic under Average only, Google's Gemini 4 Argon under Max only, and OpenAI, Alibaba and MiniMax under Not stated. OSWorld-Verified: Anthropic, Google, Meta and NVIDIA under Average only, Alibaba and MiniMax under Not stated. Terminal-Bench 4.1: Google under Average only. Terminal-Bench Science 0.1: Anthropic and Google under Average only, OpenAI, Moonshot AI and Tencent under Not stated. SEC-bench Pro: Google under Average only. SWE-bench Verified: Anthropic, Google, OpenAI, Meta, Meituan, Poolside and MiniMax under Average only, Anthropic's Claude Opus 4, Moonshot AI's Kimi K2 and NVIDIA's Nemotron 3.5 Super VL under Both, and Alibaba, Z.ai, Tencent, DeepSeek, Mistral and Ricursive under Not stated.
Google's Gemini 4 Hydrogen methodology of Nov 12, 2026 reports OSWorld 2.1 “maxed over 3 runs”. It averages Terminal-Bench 4.1 and Terminal-Bench Science 0.1 over 5 attempts, and Anthropic averages OSWorld 2.1 over 5. Each lab is placed by what its own document says. Download

Google reports its Gemini 4 models' OSWorld 2.1 scores as the maximum over three runs. Its evaluation methodology for Gemini 4 Hydrogen, published with the model on Nov 12, says the scores are “self-computed, maxed over 3 runs with a single attempt per run”. Anthropic reports the same benchmark as an average of five attempts.

Google averages its Terminal-Bench runs

The same document states a general rule: “To reduce variance, we average over multiple trials for smaller benchmarks.” It reports Terminal-Bench 4.1 and Terminal-Bench Science 0.1 “averaged over 5 attempts per task”. The Gemini 4 Carbon methodology reports SEC-bench Pro “averaged over 3 independent attempts per task”. It uses the same “maxed over 3 runs” sentence for Gemini 4 Carbon and Gemini 4 Argon on OSWorld 2.1.

Google averaged on OSWorld-Verified, the version before OSWorld 2.0. Its methodology for Gemini 3.6 Flash on Jul 21 said the OSWorld-Verified scores were “averaged over 5 runs with a single attempt per run”.

The maximum arrived with OSWorld 2.0. Google's methodology for Gemini 3.7 Flash, a model it launched on Aug 13, reports OSWorld 2.0 “maxed over 3 runs with a single attempt per run”. Its documents for Gemini 3.8 Flash and Gemini 4 Argon repeat the sentence. Google's documents don't say whether the maximum is the best of three whole runs or the best result on each task. The Gemini 4 Hydrogen document applies the same sentence to the Claude Sonnet 5.5 score in its table, which it says Google computed itself. The table prints 83.9% for Claude Sonnet 5.5. Anthropic's Claude Haiku 5.5 system card gives the same 83.9% for Claude Sonnet 5.5 as an average of five attempts.

Anthropic averages five attempts on the same tasks

Anthropic reports OSWorld 2.1 on the same offline set of 82 tasks. Its Claude Fable 5.2 system card on Nov 9 gives each score “as Pass@1 averaged over five independent attempts per task”. Google's Gemini 4 Carbon table prints 87.2% for Claude Opus 5.5 in the same row as 81.5% for Gemini 4 Carbon. Anthropic's Claude Haiku 5.5 system card reports that 87.2% as an average of five attempts.

Four labs don't say how they combined runs

OpenAI, Meta, Moonshot AI and Tencent each report an OSWorld 2.1 score. OpenAI’s GPT-6.1 Terra post, Meta’s Muse Ember evaluation report, Moonshot AI’s Kimi K3.1 Pro post and Tencent’s Hy4 post don't say how repeated runs were combined. The benchmark's maker doesn't set a rule in its OSWorld 2.1 release notes.

Three labs printed a pick from several attempts

On SWE-bench Verified, three labs have printed a second score that draws on several attempts. Anthropic’s Claude 4 post on May 22, 2025 gave Claude Opus 4 a score “averaged over 10 trials, single-attempt patches”. It added a second score from “sampling multiple sequences and selecting the single best via an internal scoring model”. Moonshot AI’s Kimi K2 post on Jul 11, 2025 printed a “Single Attempt” row and a “Multiple Attempts” row, and its model card describes the second in the same words as Anthropic. NVIDIA’s Nemotron 3.5 technical report gives a score “averaged over three runs” and a second from three attempts, where “the same checkpoint then serves as a verifier, selecting one candidate”.

Moonshot AI also reports a maximum alone, on a benchmark outside the table. Moonshot AI’s Kimi K3.1 Pro post on Oct 24 reports ExploitBench as “the pass@5 max capability score: for each environment, the best run across 5 seeds”.

How we sorted the labs

A lab is under Average only where its own document says the score is averaged over runs, attempts or trials. The row names its latest model whose document says so. For Not stated, we read the document that reports the lab's score, its table footnotes and its methodology section. A lab is listed there when that document doesn't say how runs were combined. We left out cases a reader could take two ways, such as a metric labelled Pass@1 with no word on repeats. We also left out scores a lab took from another party's run. Inferring OpenAI's and Moonshot AI's Unstated OSWorld 2.1 Setups covers the task set and scoring behind the same scores.

BenchmarkLab, modelReportedThe lab's words
OSWorld 2.1Anthropic, Claude Fable 5.2Average only“averaged over five independent attempts per task”
OSWorld 2.1Google, Gemini 4 HydrogenMax only“maxed over 3 runs with a single attempt per run”
OSWorld 2.1OpenAI, GPT-6.1 TerraNot statedDoesn't say
OSWorld 2.1Meta, Muse EmberNot statedDoesn't say
OSWorld 2.1Moonshot AI, Kimi K3.1 ProNot statedDoesn't say
OSWorld 2.1Tencent, Hy4Not statedDoesn't say
OSWorld 2.0Anthropic, Claude Opus 5.5Average only“each as pass@1 averaged over five independent runs”
OSWorld 2.0Google, Gemini 4 ArgonMax only“maxed over 3 runs with a single attempt per run”
OSWorld 2.0OpenAI, GPT-6.1 SolNot statedDoesn't say
OSWorld 2.0Alibaba, Qwen3.8-MaxNot statedDoesn't say
OSWorld 2.0MiniMax, MiniMax M3.1 FlashNot statedDoesn't say
OSWorld-VerifiedAnthropic, Claude Sonnet 5Average only“first-attempt success rate, averaged over five runs”
OSWorld-VerifiedGoogle, Gemini 3.6 FlashAverage only“averaged over 5 runs with a single attempt per run”
OSWorld-VerifiedMeta, Muse Glimmer 30BAverage only“averaged over four attempts per task”
OSWorld-VerifiedNVIDIA, Nemotron 3.5 Super VLAverage only“We report the average task score, which includes partial credit, over three runs.”
OSWorld-VerifiedAlibaba, Qwen3.8-MaxNot statedDoesn't say
OSWorld-VerifiedMiniMax, MiniMax M3Not statedDoesn't say
Terminal-Bench 4.1Google, Gemini 4 HydrogenAverage only“averaged over 5 attempts per task”
Terminal-Bench Science 0.1Anthropic, Claude Fable 5.2Average only“Scores are averaged over 10 trials per task”
Terminal-Bench Science 0.1Google, Gemini 4 HydrogenAverage only“averaged over 5 attempts per task”
Terminal-Bench Science 0.1OpenAI, GPT-6.1 TerraNot statedDoesn't say
Terminal-Bench Science 0.1Moonshot AI, Kimi K3.1 ProNot statedDoesn't say
Terminal-Bench Science 0.1Tencent, Hy4Not statedDoesn't say
SEC-bench ProGoogle, Gemini 4 CarbonAverage only“averaged over 3 independent attempts per task”
SWE-bench VerifiedAnthropic, Claude Opus 5Average only“each score is an average over five trials”
SWE-bench VerifiedGoogle, Gemini 3.1 ProAverage only“Averaged over 10x runs for SWE-Bench Verified”
SWE-bench VerifiedOpenAI, GPT-5Average only“we average over 4 tries per instance”
SWE-bench VerifiedMeta, Muse Glimmer 30BAverage only“we average results across 4 runs”
SWE-bench VerifiedMeituan, LongCat-Flash-Thinking-2601Average only“SWE-bench Verified (Avg@5)”
SWE-bench VerifiedPoolside, Laguna XS 2.1Average only“mean pass@1 averaged over 4 attempts per task”
SWE-bench VerifiedMiniMax, MiniMax M3Average only“Each test was run 4 times and the average was taken.”
SWE-bench VerifiedNVIDIA, Nemotron 3.5 Super VLBoth“we report the resolve rate averaged over three runs” and “The same checkpoint then serves as a verifier, selecting one candidate”
SWE-bench VerifiedAnthropic, Claude Opus 4Both“averaged over 10 trials, single-attempt patches” and “selecting the single best via an internal scoring model”
SWE-bench VerifiedMoonshot AI, Kimi K2Both“Single Attempt (Acc)” and “Multiple Attempts (Acc)”, the second by “selecting the single best via an internal scoring model”
SWE-bench VerifiedAlibaba, Qwen3.7-PlusNot statedDoesn't say
SWE-bench VerifiedZ.ai, GLM-5Not statedDoesn't say
SWE-bench VerifiedTencent, Hy3Not statedDoesn't say
SWE-bench VerifiedDeepSeek, DeepSeek-V4-ProNot statedDoesn't say
SWE-bench VerifiedMistral, Mistral Medium 3.5Not statedDoesn't say
SWE-bench VerifiedRicursive, Ripple-65B PreviewNot statedDoesn't say

OSWorld 2.1, offline set, partial score

ModelScoreSources
Claude Opus 5.5Claude Opus 5.5, mean of 5 attempts87.2%Claude Haiku 5.5 System Cardsection 8.9.3 OSWorld 2.1 offline subset, text (PDF page 128), the Claude Opus 5.5 partial score
Claude Sonnet 5.5Claude Sonnet 5.5, mean of 5 attempts83.9%Introducing Claude Haiku 5.5benchmark table (HTML), row 'Computer use / OSWorld 2.1', column 'Sonnet 5.5', the value tagged 'Offline subset'
Gemini 4 CarbonGemini 4 Carbon, max of 3 runs81.5%Announcementbenchmark table (image), row 'OSWorld-2.1', 'Offline subset', column 'Gemini 4 Carbon'

Sources

Models

Related

Tags