{
  "api_version": "1",
  "updated": "2026-11-10",
  "latest_event": "2026-11-09",
  "rights": "© Frontier Log, all rights reserved.",
  "story": {
    "slug": "labs-run-benchmarks-different-ways",
    "title": "AI Labs Run AutomationBench and OSWorld in Different Ways",
    "description": "Labs that report AutomationBench and OSWorld 2.1 choose different task sets and scoring rules. Only Anthropic reports all four OSWorld setups for one model.",
    "url": "https://frontierlog.ai/stories/labs-run-benchmarks-different-ways",
    "date": "2026-11-08",
    "image": "https://frontierlog.ai/og/story-labs-run-benchmarks-different-ways.png",
    "event": null,
    "tags": [
      "agents"
    ],
    "steps": [],
    "tables": [
      {
        "title": "AutomationBench, private set, strict",
        "unit": "%",
        "rows": [
          {
            "event": "2026-09-29-gpt-6-1-sol",
            "date": "2026-09-29",
            "provider": "openai",
            "value": 36.1,
            "printed": "36.1%",
            "source": "https://openai.com/index/introducing-gpt-6-1-sol",
            "where": "AutomationBench chart under \"Professional work\" (score vs cost per task; no value labels): point 'GPT-6.1 Sol', effort 'Max', read from the point's aria-label (load the page in a browser and hover or inspect the SVG point)"
          },
          {
            "event": "2026-09-22-claude-opus-5-5",
            "date": "2026-09-22",
            "provider": "anthropic",
            "value": 40,
            "printed": "40.0%",
            "source": "https://www.anthropic.com/claude-opus-5-5",
            "where": "benchmark table, row 'AutomationBench', column 'Opus 5.5'"
          },
          {
            "event": "2026-10-28-gemini-4-carbon",
            "date": "2026-10-28",
            "provider": "google",
            "value": 53.9,
            "printed": "53.9%",
            "source": "https://blog.google/innovation-and-ai/models-and-research/gemini-models/introducing-gemini-4-carbon/",
            "where": "benchmark table (image), row 'AutomationBench', 'Zapier', column 'Gemini 4 Carbon'"
          }
        ]
      },
      {
        "title": "AutomationBench, private set, partial credit (Artificial Analysis)",
        "unit": "%",
        "rows": [
          {
            "event": "2026-10-24-kimi-k3-1-pro",
            "date": "2026-10-24",
            "provider": "moonshot",
            "value": 73.3,
            "printed": "73.3",
            "source": "https://www.kimi.ai/blog/kimi-k3-1-pro",
            "where": "'Benchmarks' table (rendered by the page's script), section 'Agentic', row 'Automation Bench-AA', column 'Kimi K3.1 Pro (max)'; runner: the table's notes, 'AutomationBench-AA scores are cited from https://artificialanalysis.ai/'"
          },
          {
            "event": "2026-10-06-mistral-large-4",
            "date": "2026-10-06",
            "provider": "mistral",
            "value": 59.9,
            "printed": "59.9",
            "source": "https://mistral.ai/news/mistral-large-4/",
            "where": "'Agentic Workflows' section, chart tab 'AA - AutomationBench' (image titled 'Artificial Analysis - AutomationBench'), value label on the 'Mistral Large 4 Preview' bar; the section's text gives the same score for AutomationBench, '657 business workflows'"
          },
          {
            "event": "2026-11-05-inkling-2-small",
            "date": "2026-11-05",
            "provider": "thinking-machines",
            "value": 45.6,
            "printed": "45.6%",
            "source": "https://thinkingmachines.ai/model-card/inkling-2-small/",
            "where": "section '5. Evaluations', evaluations table, Enterprise Operations group, row 'AutomationBench v1.0.6', column 'Inkling 2 Small' (first model column); set and runner: launch post thinkingmachines.ai/news/inkling-2-small/, list of externally reported evals right after its evaluations table, before 'Limitations'"
          }
        ]
      },
      {
        "title": "AutomationBench, public set, strict",
        "unit": "%",
        "rows": [
          {
            "event": "2026-11-04-hy4",
            "date": "2026-11-04",
            "provider": "tencent",
            "value": 56.2,
            "printed": "56.2",
            "source": "https://hy.tencent.ai/research/hy4",
            "where": "'Appendix: benchmark results' table image, row 'AutomationBench v1.0.6', column 'Hy4'"
          },
          {
            "event": "2026-09-10-deepseek-v4-1-flash",
            "date": "2026-09-10",
            "provider": "deepseek",
            "value": 54.8,
            "printed": "54.8",
            "source": "https://api-docs.deepseek.com/news/news260910/",
            "where": "benchmark table image (second image), row 'Automation-Bench', column 'DeepSeek V4.1-Flash' (the first chart image, v4.1_260910_benchmark.png, prints the same value label)"
          },
          {
            "event": "2026-11-01-longcat-2-5",
            "date": "2026-11-01",
            "provider": "meituan",
            "value": 51.8,
            "printed": "51.8",
            "source": "https://longcat.ai/blog/longcat-2.5",
            "where": "'Evaluations' section table at the end of the post (rendered by the page's script; load https://longcat.ai/blog/longcat-2.5 in a browser and scroll to the end), row 'AutomationBench v1.0.6' under 'General Agent', column 'LongCat-2.5'"
          }
        ]
      },
      {
        "title": "OSWorld 2.1",
        "unit": "%",
        "rows": [
          {
            "event": "2026-09-22-claude-opus-5-5",
            "date": "2026-09-22",
            "provider": "anthropic",
            "value": 81.8,
            "printed": "81.8%",
            "source": "https://www.anthropic.com/claude-opus-5-5",
            "where": "benchmark table (HTML), row 'Computer use / OSWorld 2.1', column 'Opus 5.5', the value tagged 'partial'"
          },
          {
            "event": "2026-09-22-claude-opus-5-5",
            "date": "2026-09-22",
            "provider": "anthropic",
            "value": 48.7,
            "printed": "48.7%",
            "source": "https://www.anthropic.com/claude-sonnet-5-5-system-card",
            "where": "section 8.13.3 OSWorld 2.1, text (PDF page 130), the Claude Opus 5.5 strict pass rate"
          },
          {
            "event": "2026-09-22-claude-opus-5-5",
            "date": "2026-09-22",
            "provider": "anthropic",
            "value": 87.2,
            "printed": "87.2%",
            "source": "https://www.anthropic.com/claude-haiku-5-5-system-card",
            "where": "section 8.9.3 OSWorld 2.1 offline subset, text (PDF page 128), the Claude Opus 5.5 partial score"
          },
          {
            "event": "2026-09-22-claude-opus-5-5",
            "date": "2026-09-22",
            "provider": "anthropic",
            "value": 53.2,
            "printed": "53.2%",
            "source": "https://www.anthropic.com/claude-haiku-5-5-system-card",
            "where": "section 8.9.3 OSWorld 2.1 offline subset, text (PDF page 128), the Claude Opus 5.5 strict pass rate"
          },
          {
            "event": "2026-10-12-muse-wonder",
            "date": "2026-10-12",
            "provider": "meta",
            "value": 72.6,
            "printed": "72.6",
            "source": "https://research.meta.ai/blog/muse-wonder",
            "where": "evaluation table (image), row 'OSWorld V2.1', column 'Muse Wonder (max)', the value marked '(partial)'"
          },
          {
            "event": "2026-10-12-muse-wonder",
            "date": "2026-10-12",
            "provider": "meta",
            "value": 34.3,
            "printed": "34.3",
            "source": "https://research.meta.ai/blog/muse-wonder",
            "where": "evaluation table (image), row 'OSWorld V2.1', column 'Muse Wonder (max)', the value marked '(binary)'"
          },
          {
            "event": "2026-11-04-hy4",
            "date": "2026-11-04",
            "provider": "tencent",
            "value": 78.9,
            "printed": "78.9",
            "source": "https://hy.tencent.ai/research/hy4",
            "where": "'Appendix: benchmark results' table image, row 'OSWorld 2.1 (Full, partial score)', column 'Hy4'"
          },
          {
            "event": "2026-09-03-gpt-6-astra",
            "date": "2026-09-03",
            "provider": "openai",
            "value": 80.9,
            "printed": "80.9%",
            "source": "https://openai.com/index/introducing-gpt-6-1-terra",
            "where": "\"OSWorld 2.1, offline set\" chart (score vs cost per task; no value labels): point 'GPT-6 Astra', effort 'Max', read from the point's aria-label (load the page in a browser and hover or inspect the SVG point)"
          },
          {
            "event": "2026-10-28-gemini-4-carbon",
            "date": "2026-10-28",
            "provider": "google",
            "value": 81.5,
            "printed": "81.5%",
            "source": "https://blog.google/innovation-and-ai/models-and-research/gemini-models/introducing-gemini-4-carbon/",
            "where": "benchmark table (image), row 'OSWorld-2.1', 'Offline subset', column 'Gemini 4 Carbon'"
          }
        ]
      }
    ],
    "cites": [
      {
        "url": "https://zapier.com/benchmarks",
        "by": "Zapier",
        "medium": "AutomationBench leaderboard, read Nov 8, 2026",
        "date": "2026-11-08"
      },
      {
        "url": "https://github.com/zapier/AutomationBench/blob/4a8e106/README.md",
        "by": "Zapier",
        "medium": "AutomationBench README, release 1.0.6",
        "date": "2026-08-04"
      },
      {
        "url": "https://artificialanalysis.ai/methodology/intelligence-benchmarking",
        "by": "Artificial Analysis",
        "medium": "methodology page, read Nov 8, 2026",
        "date": "2026-11-08"
      },
      {
        "url": "https://deepmind.google/models/evals-methodology/gemini-4-carbon",
        "by": "Google",
        "medium": "evaluation methodology",
        "date": "2026-10-28"
      }
    ],
    "sources": [
      "https://zapier.com/benchmarks",
      "https://github.com/zapier/AutomationBench/blob/4a8e106/README.md",
      "https://openai.com/index/introducing-gpt-6-1-sol",
      "https://www.anthropic.com/claude-opus-5-5",
      "https://blog.google/innovation-and-ai/models-and-research/gemini-models/introducing-gemini-4-carbon/",
      "https://hy.tencent.ai/research/hy4",
      "https://longcat.ai/blog/longcat-2.5",
      "https://research.meta.ai/blog/introducing-muse-spark-1-3",
      "https://qwen.ai/blog?id=qwen3.8",
      "https://artificialanalysis.ai/methodology/intelligence-benchmarking",
      "https://www.kimi.ai/blog/kimi-k3-1-pro",
      "https://mistral.ai/news/mistral-large-4/",
      "https://thinkingmachines.ai/model-card/inkling-2-small/",
      "https://research.meta.ai/blog/muse-wonder",
      "https://www.anthropic.com/claude-sonnet-5-5-system-card",
      "https://www.anthropic.com/claude-haiku-5-5-system-card",
      "https://deepmind.google/models/evals-methodology/gemini-4-carbon",
      "https://openai.com/index/introducing-gpt-6-1-terra",
      "https://api-docs.deepseek.com/news/news260910/",
      "https://www.minimax.io/blog/minimax-m31f",
      "https://www.ricursive.com/ripple",
      "https://z.ai/blog/glm-5.3",
      "https://huggingface.co/deepseek-ai/DeepSeek-V4.1-Flash/blob/main/DeepSeek_V41_Tech_Report.pdf"
    ],
    "updates": [],
    "body": [
      {
        "figure": "main"
      },
      {
        "p": "When two labs report the same benchmark, they often ran it in different ways. On [AutomationBench](https://frontierlog.ai/benchmark/automationbench), Zapier's test of AI agents on business workflows, the labs' figures come from two task sets, scored in two ways. On [OSWorld 2.1](https://frontierlog.ai/benchmark/osworld-2-1), a test of agents using a computer, they come from a full set and an offline subset, each scored two ways. Of the labs we cover, only Anthropic reports all four OSWorld setups for one model."
      },
      {
        "h": "Three setups on AutomationBench"
      },
      {
        "p": "Zapier scores its [leaderboard](https://zapier.com/benchmarks) on a private task set it doesn't release, and counts a task as passed only when it is completed correctly. Its [README](https://github.com/zapier/AutomationBench/blob/4a8e106/README.md) says the private set \"is purposely harder\" than the 600 public tasks. Anthropic's [Claude Sonnet 5.5 system card](https://www.anthropic.com/claude-sonnet-5-5-system-card) says its results are on \"Zapier's private evaluation set\", and Claude Opus 5.5 scored 40.0% [there](https://www.anthropic.com/claude-opus-5-5). Google's figure of 53.9% for [Gemini 4 Carbon](https://blog.google/innovation-and-ai/models-and-research/gemini-models/introducing-gemini-4-carbon/) uses the private set, its methodology says. OpenAI's [GPT-6.1 Sol post](https://openai.com/index/introducing-gpt-6-1-sol) links its AutomationBench figures to Zapier's leaderboard and gives 36.1% at max effort."
      },
      {
        "p": "Tencent, DeepSeek and Meituan report their own runs on the public tasks. Tencent's [Hy4](https://hy.tencent.ai/research/hy4) scored 56.2%, DeepSeek's [V4.1-Flash](https://api-docs.deepseek.com/news/news260910/) 54.8% and Meituan's [LongCat-2.5](https://longcat.ai/blog/longcat-2.5) 51.8%. DeepSeek gives the figure as pass@1, the share of tasks passed on one attempt, and names the public set in its [technical report](https://huggingface.co/deepseek-ai/DeepSeek-V4.1-Flash/blob/main/DeepSeek_V41_Tech_Report.pdf)."
      },
      {
        "p": "Moonshot AI, Mistral and Thinking Machines print Artificial Analysis's version instead. Its [methodology](https://artificialanalysis.ai/methodology/intelligence-benchmarking) says it runs a private 657-task split. A task earns the share of its objectives completed, or 0 if the model breaks a guardrail or the task errors, so a task can earn partial credit. [Kimi K3.1 Pro](https://www.kimi.ai/blog/kimi-k3-1-pro) scored 73.3% there, [Mistral Large 4](https://mistral.ai/news/mistral-large-4/) 59.9% and [Inkling 2 Small](https://thinkingmachines.ai/model-card/inkling-2-small/) 45.6%."
      },
      {
        "h": "Four setups on OSWorld 2.1"
      },
      {
        "p": "OSWorld 2.1 gives a partial score for each task and a strict pass rate, on its full task set or on an 82-task subset that runs with no internet access. Anthropic reports Claude Opus 5.5 in all four: 81.8% on the full set with partial credit and 48.7% strict, 87.2% on the offline subset with partial credit and 53.2% strict. Meta reports [Muse Wonder](https://research.meta.ai/blog/muse-wonder) on 108 tasks, 72.6% with partial credit and 34.3% strict. Tencent reports Hy4 at 78.9% with partial credit."
      },
      {
        "p": "OpenAI and Google report only the offline subset. OpenAI's [GPT-6.1 Terra post](https://openai.com/index/introducing-gpt-6-1-terra) charts GPT-6 Astra at 80.9% at max effort without naming the scoring. Its [GPT-6.1 Sol post](https://openai.com/index/introducing-gpt-6-1-sol) says of the earlier OSWorld 2.0 that \"we report the partial reward on the offline set\". Google's [Gemini 4 Carbon post](https://blog.google/innovation-and-ai/models-and-research/gemini-models/introducing-gemini-4-carbon/) prints the same 80.9% for GPT-6 Astra on its offline-subset row, next to 81.5% for Gemini 4 Carbon. Its [methodology](https://deepmind.google/models/evals-methodology/gemini-4-carbon) says it reports \"the partial score from the offline subset\"."
      },
      {
        "h": "What the matrix leaves out"
      },
      {
        "p": "The image marks the task set and scoring rule behind each lab's figure, as the lab's post, the methodology it cites, the benchmark's own scoring or another lab's report of the same figure states them. Each column is a setting at least two labs report, and every AutomationBench figure is one its source, or the leaderboard or methodology it draws on, ties to release v1.0.6. Other settings change a score too: the agent harness, the effort setting, the number of runs and fallback models. The image leaves them out."
      },
      {
        "p": "Some labs report a figure without enough detail to place it. [MiniMax](https://www.minimax.io/blog/minimax-m31f) and [Ricursive](https://www.ricursive.com/ripple) name the AutomationBench public set but not the scoring. Meta's [Muse Spark 1.3](https://research.meta.ai/blog/introducing-muse-spark-1-3) figure, on \"the public v3 task set\", and Alibaba's [Qwen3.8-Max](https://qwen.ai/blog?id=qwen3.8) figure don't name an AutomationBench release. [Z.ai](https://z.ai/blog/glm-5.3) doesn't name a set and says it incorporated \"the fix for the null-type handling issue\". Moonshot AI reports [Kimi K3.1 Pro](https://www.kimi.ai/blog/kimi-k3-1-pro) on OSWorld 2.1 without naming the set or the scoring."
      }
    ],
    "events": [
      "2026-08-03-qwen3-8-max",
      "2026-08-14-glm-5-3",
      "2026-09-02-muse-spark-1-3",
      "2026-09-10-deepseek-v4-1-flash",
      "2026-09-22-claude-opus-5-5",
      "2026-09-29-gpt-6-1-sol",
      "2026-10-06-mistral-large-4",
      "2026-10-12-muse-wonder",
      "2026-10-19-minimax-m3-1-flash",
      "2026-10-24-kimi-k3-1-pro",
      "2026-10-26-gpt-6-1-terra",
      "2026-10-28-gemini-4-carbon",
      "2026-11-01-longcat-2-5",
      "2026-11-04-hy4",
      "2026-11-04-ripple-65b-preview"
    ],
    "stories": [
      "automationbench-private-and-public-sets",
      "who-runs-automationbench"
    ],
    "models": [
      "claude-opus-5-5",
      "gpt-6-1-sol",
      "gemini-4-carbon",
      "kimi-k3-1-pro",
      "mistral-large-4",
      "inkling-2-small",
      "hy4",
      "longcat-2-5",
      "muse-spark-1-3",
      "qwen3-8-max",
      "muse-wonder",
      "gpt-6-astra",
      "deepseek-v4-1-flash"
    ]
  }
}