Frontier Log

Benchmark · Active

AutomationBench, no release named

Created by Zapier, 20 Apr 2026 · Higher is better · Website

0102030405060Apr 2026Jul 2026Oct 202649.6%Muse Spark 1.336.4%Ripple-65B Preview25.7%DeepSeek-V4-Flash-Vision-Exp38.2%Muse Spark 1.227.3%Qwen3.8-Max30.8%Kimi K314.2%Qwen3.7-Max12.8%DeepSeek-V4-Pro0102030405060Apr 2026Jul 2026Oct 202649.6%Muse Spark 1.336.4%Ripple-65B Preview25.7%DeepSeek-V4-Flash-Vision-Exp38.2%Muse Spark 1.230.8%Kimi K3

Results

ModelScoreSource
Muse Spark 1.32 Sep 2026 · Meta49.6%Introducing Muse Spark 1.3Checked A person checked this number against its source on 20 Oct 2026. This confirms the number matches what the lab published. It doesn't confirm that the score was reproduced.max effort; pass@1; the post's evaluation methodology names the "public v3 task set"Source image
Muse Spark 1.25 Aug 2026 · Meta38.2%Introducing Muse Spark 1.3Checked A person checked this number against its source on 20 Oct 2026. This confirms the number matches what the lab published. It doesn't confirm that the score was reproduced.xhigh effort; pass@1; the post's evaluation methodology names the "public v3 task set"; reported laterSource image
Ripple-65B Preview4 Nov 2026 · Ricursive Intelligence36.4%Introducing Ripple-65B Preview, a Frontier Open-Weight ModelCorroborated The same number is printed in 1 other lab document: 1OpenCode harness and maximum effort per the model cardOur capture
DeepSeek-V4-Pro-081313 Aug 2026 · DeepSeek31.8%DeepSeek-V4-Pro GA ReleaseChecked A person checked this number against its source on 20 Oct 2026. This confirms the number matches what the lab published. It doesn't confirm that the score was reproduced.Source image
DeepSeek-V4-Pro-081313 Aug 2026 · DeepSeek43.2%DeepSeek-V4.1-Flash: Smarter, Faster, More EfficientCorroborated The same number is printed in 1 other lab document: 1v1.0.6; set named in the technical report the post links, which prints the same figure; reported later; another report, not chartedSource image
Kimi K316 Jul 2026 · Moonshot AI30.8%Kimi K3: Open Frontier IntelligenceChecked A person checked this number against its source on 20 Oct 2026. This confirms the number matches what the lab published. It doesn't confirm that the score was reproduced.max reasoning effort; release not named, post predates release 1.0.6Our capture
Qwen3.8-Max3 Aug 2026 · Alibaba27.3%Qwen3.8-Max: A New Bar for Coding and CoworkChecked A person checked this number against its source on 19 Oct 2026. This confirms the number matches what the lab published. It doesn't confirm that the score was reproduced.pass@1Our capture
DeepSeek-V4-Flash-Vision-Exp21 Aug 2026 · DeepSeek25.7%DeepSeek-V4-Flash-Vision-Exp Release: Multimodal API Now LiveChecked A person checked this number against its source on 20 Oct 2026. This confirms the number matches what the lab published. It doesn't confirm that the score was reproduced.DeepSeek Harness minimal modeSource image
Qwen3.7-Max20 May 2026 · Alibaba14.2%Qwen3.8-Max: A New Bar for Coding and CoworkChecked A person checked this number against its source on 19 Oct 2026. This confirms the number matches what the lab published. It doesn't confirm that the score was reproduced.pass@1; reported laterOur capture
DeepSeek-V4-Pro24 Apr 2026 · DeepSeek12.8%DeepSeek-V4-Pro GA ReleaseChecked A person checked this number against its source on 20 Oct 2026. This confirms the number matches what the lab published. It doesn't confirm that the score was reproduced.reported laterSource image

All benchmarks