SWE-bench Pro
- OpenAI
- Anthropic
- DeepSeek
- Alibaba
- Moonshot AI
- MiniMax
- Thinking Machines
- Poolside
- Reported in a later post or card
Results
| Model | Score | Source | Human-verified |
|---|---|---|---|
| Claude Haiku 5.57 Oct 2026 · Anthropic | 64.8% | Claude Haiku 5.5 System Cardadaptive thinking, max effort, average of 5 trialsThe source doesn't say whether this is the public or the commercial set | Yes |
| Claude Sonnet 5.528 Sep 2026 · Anthropic | 81.3% | Claude Sonnet 5.5 System Cardadaptive thinking, max effort, average of 5 trialsThe source doesn't say whether this is the public or the commercial set | Yes |
| Claude Opus 5.522 Sep 2026 · Anthropic | 89.9% | Claude Opus 5.5 System Cardadaptive thinking, max effort, average of 5 trialsThe source doesn't say whether this is the public or the commercial set | Yes |
| Claude Fable 5.11 Sep 2026 · Anthropic | 81.2% | Claude Fable 5.1 and Claude Mythos 5.1 System Cardaverage of 5 trials; from the card's prose (its table merges Fable 5.1 and Mythos 5.1)The source doesn't say whether this is the public or the commercial set | Yes |
| Qwen3.8-Max3 Aug 2026 · Alibaba | 67.7% | Qwen3.8-Max: A New Bar for Coding and CoworkClaude Code harnessRun on the lab's own corrected version of the public set | Yes |
| Claude Opus 524 Jul 2026 · Anthropic | 79.2% | Claude Opus 5 System Cardadaptive thinking, max effort, average of 5 trialsThe source doesn't say whether this is the public or the commercial set | Yes |
| Gemini 3.6 Flash21 Jul 2026 · Google | 58.7% | Gemini 3.6 Flash - Model Cardresults as of July 2026 | Yes |
| Inkling15 Jul 2026 · Thinking Machines | 54.3% | Inkling: Our Open-Weights Modeleffort 0.99, temperature 1.0, 256K-token trajectory limit | Yes |
| GPT-5.6 Sol9 Jul 2026 · OpenAI | 64.6% | GPT-5.6: Frontier intelligence that scales with your ambitionThe source doesn't say whether this is the public or the commercial set | Yes |
| Laguna XS 2.12 Jul 2026 · Poolside | 47.6% | Introducing Laguna XS 2.1poolside harness on Harbor, mean pass@1 over 2 attempts, some task images patched for infrastructure reliabilityThe source doesn't say whether this is the public or the commercial set | Yes |
| Claude Sonnet 530 Jun 2026 · Anthropic | 63.2% | Introducing Claude Sonnet 5The source doesn't say whether this is the public or the commercial setSource image | Yes |
| Claude Fable 59 Jun 2026 · Anthropic | 80% | Claude Fable 5 and Claude Mythos 5 System Cardadaptive thinking, max effort, average of 5 trials; with production safeguards, including fallback to Opus 4.8The source doesn't say whether this is the public or the commercial set | Yes |
| Claude Mythos 59 Jun 2026 · Anthropic | 80.3% | Claude Fable 5 and Claude Mythos 5 System Cardadaptive thinking, max effort, average of 5 trialsThe source doesn't say whether this is the public or the commercial set | Yes |
| MiniMax M31 Jun 2026 · MiniMax | 59.0% | MiniMax M3: Frontier Coding, 1M Context, Native Multimodality — All in One ModelClaude Code scaffoldThe source doesn't say whether this is the public or the commercial setSource image | Yes |
| Claude Opus 4.828 May 2026 · Anthropic | 69.2% | Introducing Claude Opus 4.8The source doesn't say whether this is the public or the commercial setSource image | Yes |
| Gemini 3.5 Flash19 May 2026 · Google | 55.1% | Gemini 3.5: frontier intelligence with actionsingle attemptSource image | Yes |
| DeepSeek-V4-Pro24 Apr 2026 · DeepSeek | 55.4% | DeepSeek V4 Preview ReleaseMax reasoning effortThe source doesn't say whether this is the public or the commercial setSource image | Yes |
| GPT-5.523 Apr 2026 · OpenAI | 58.6% | Introducing GPT-5.5 | Yes |
| Kimi K2.620 Apr 2026 · Moonshot AI | 58.6% | Kimi K2.6: Advancing Open-Source Codingin-house framework adapted from SWE-agent, thinking mode, average of 10 runsThe source doesn't say whether this is the public or the commercial set | Yes |
| Claude Opus 4.716 Apr 2026 · Anthropic | 64.3% | Introducing Claude Opus 4.7memorization screens flag a subset of problems; the margin over Opus 4.6 holds without themThe source doesn't say whether this is the public or the commercial setSource image | Yes |
| Claude Mythos Preview7 Apr 2026 · Anthropic | 77.8% | Introducing Claude Opus 4.7reported laterThe source doesn't say whether this is the public or the commercial setSource image | Yes |
| GPT-5.45 Mar 2026 · OpenAI | 57.7% | Introducing GPT-5.4 | Yes |
| Gemini 3.1 Pro19 Feb 2026 · Google | 54.2% | Gemini 3.1 Pro: A smarter model for your most complex taskssingle attempt, Thinking (High)Source image | Yes |
| MiniMax M2.512 Feb 2026 · MiniMax | 55.4% | MiniMax M2.5: Built for Real-World Productivity.Claude Code scaffold, average of 4 runsThe source doesn't say whether this is the public or the commercial setSource image | Yes |
| GPT-5.3-Codex5 Feb 2026 · OpenAI | 56.8% | Introducing GPT-5.3-Codexxhigh reasoning effort | Yes |
| Kimi K2.527 Jan 2026 · Moonshot AI | 50.7% | Kimi K2.5: Visual Agentic Intelligencein-house minimal-tool framework, non-thinking mode, average of 5 runsThe source doesn't say whether this is the public or the commercial set | Yes |
| GPT-5.211 Dec 2025 · OpenAI | 55.6% | Introducing GPT-5.2 | Yes |
| Gemini 3 Pro18 Nov 2025 · Google | 43.3% | Gemini 3.1 Pro: A smarter model for your most complex tasksreported later; single attempt, Thinking (High)Source image | Yes |
Benchmark
scale.comHigher is better
Scale AIIntroduced 19 Sep 2025