Frontier Log

Tracing GPT-5.6 Sol's Unstated Task Set on Agents' Last Exam v1

OpenAI doesn't name the task set behind GPT-5.6 Sol's 53.6 on Agents' Last Exam, and its chart's costs point to all tasks on the maker's leaderboard.

A top panel, then three numbered steps with source screenshots under the heading How we trace the setup. The panel: OpenAI GPT-5.6 Sol, 53.6%. Setup in OpenAI's post: Score, task set not stated. An arrow points to a dashed box, traced and not confirmed by OpenAI: Score, all tasks. A line under the panel says the leaderboard's Score is the average partial-credit score, and its CLI subset holds the Linux command-line tasks. A key says an anchor is a figure in OpenAI's post that is also on the maker's leaderboard. Step 1, read OpenAI's chart: the chart plots GPT-5.6 Sol at 53.6% as a Score, at xhigh effort. Each point has an API cost, $248 at low effort and $519 at medium. Step 2, find the figure on the leaderboard: UC Berkeley RDI, the benchmark's maker, lists the model at 53.6% as a Score at xhigh effort on two splits, all tasks and the CLI subset. Step 3, match the cost: on all tasks, the leaderboard's costs are $248 at low effort and $519 at medium. On the CLI subset they are $132 and $279. An arrow runs from each step's number to the next.
OpenAI reports GPT-5.6 Sol at 53.6 on Agents' Last Exam as a Score and doesn't name the task set. UC Berkeley RDI's leaderboard has that Score on all tasks and the CLI subset. OpenAI's chart costs at low and medium effort equal the leaderboard's all-tasks costs. OpenAI hasn't confirmed this. Download

OpenAI reported 53.6 for GPT-5.6 Sol on Agents' Last Exam in its GPT-5.6 post on Jul 9, 2026, and called the figure "a new high". The chart under that sentence labels the figure a Score at xhigh effort. The post doesn't define the term or name the task set. The leaderboard of the benchmark's maker and the costs in OpenAI's chart point to all tasks.

The leaderboard lists 53.6% as a Score

UC Berkeley RDI makes Agents' Last Exam and publishes a leaderboard for version 1 with two measures. The page is marked as archived, and the current leaderboard covers version 1.1. The page says: "Pass Rate is the share of runs that earned a perfect score; Score is the average partial-credit score." On its Overall split, the leaderboard lists GPT-5.6 Sol on the Codex harness at xhigh effort with a Score of 53.6% and a Pass Rate of 30.6%. We read the leaderboard on Nov 13, 2026.

The same Score appears on two splits

The leaderboard has five splits. One is ALE-CLI, which it describes as "the subset of tasks that run on Linux and can be completed through command-line interaction alone". On ALE-CLI, GPT-5.6 Sol has the same Score at xhigh effort, 53.6%, and a Pass Rate of 28.6%. The model's Scores on the three other splits differ. OpenAI's headline figure matches the leaderboard's Score on Overall and on ALE-CLI, so the Score doesn't settle the task set.

The chart's costs point to all tasks

OpenAI's chart gives an API cost for each point. For GPT-5.6 Sol it shows $248 at low effort, $519 at medium, $577 at high, $763 at xhigh and $1,087 at max. The leaderboard's estimated costs on the Overall split at those efforts are $248, $519, $576, $772 and $1,084. On ALE-CLI they are $132, $279, $343, $446 and $597. The chart's costs equal the Overall costs at low and medium effort and sit within $9 of them at the other three. OpenAI's post says it estimates API cost "by looking at the production behavior of our models, and simulating offline". The leaderboard says its cost is "runner-logged cost where available, otherwise computed from token counts at list prices". From the costs we trace the 53.6 to all tasks.

The Scores in OpenAI's later posts match the Overall split too. Its GPT-6 Astra post on Sep 3 charts GPT-5.6 Sol at 45.1% at low effort, 52.1% at medium effort and 52.4% at high effort. Its GPT-6 Sol and Luna post on Sep 22 charts 52.8% at max effort. The leaderboard's Overall Scores at those four efforts are the same four figures. Its ALE-CLI Scores at those efforts are 46.8%, 53.1%, 51.2% and 50.9%. The GPT-5.6 post's own Scores sit below the leaderboard's Overall Scores. Its chart has GPT-5.6 Sol at 44.8% at low effort, 51.9% at medium effort, 52.1% at high effort and 52.7% at max effort. Its eval table prints the 52.7%.

Other labs' figures for the model

Alibaba's Qwen3.8-Max post on Aug 3 prints GPT-5.6 Sol as "30.6 / 53.6" in a row labelled "Agents' Last Exam (Pass / Score)". Those are the leaderboard's Pass Rate and Score on the Overall split at xhigh effort. Alibaba's column is headed "GPT5.6 Sol (max)", and the leaderboard's max-effort row has 29.6% and 52.8%.

Z.ai's GLM-5.3 post on Aug 14 prints 28.6 for GPT-5.6 Sol in a row labelled "Agents' Last Exam" and "ALE-CLI". The post doesn't name the measure. Its footnote describes Z.ai's own run of the 105 tasks with the Claude Code harness at max effort. The leaderboard's ALE-CLI Pass Rate for GPT-5.6 Sol at xhigh effort is 28.6%. DeepSeek's V4.1 tech report on Sep 10 prints 26.7 for GPT-5.6 Sol in a row labelled "Agents' Last Exam(Pass@1)", under a column headed "Max". The report says it evaluates on ALE-CLI. The leaderboard's ALE-CLI Pass Rate at max effort is 26.7%.

How we trace a setup

We trace a setup by matching the lab's figures to the leaderboard of the benchmark's maker, where the measure and the task set are stated. The leaderboard doesn't say who ran each row. OpenAI's post links the benchmark's site and doesn't say the figure comes from its leaderboard. OpenAI's GPT-6 Astra post says its "GPT evaluations were run in our research environment or via our API". OpenAI hasn't confirmed the setup we trace. Inferring GPT-5.6 Sol's Unstated Setup on OSWorld 2.0 traces another of the model's results.

Agents' Last Exam v1, GPT-5.6 Sol in OpenAI's posts

ModelScoreSources
GPT-5.6 SolGPT-5.6 Sol53.6%Announcementopening section, paragraph beginning 'We trained GPT-5.6 to get more useful work from every token', the GPT-5.6 Sol figure on Agents' Last Exam
GPT-5.6 SolGPT-5.6 Sol, in the post's eval table52.7%Announcementprofessional eval table (first table, columns GPT-5.6 Sol, Terra, Luna, GPT-5.5, ...), row 'Agents' Last Exam', column 'GPT-5.6 Sol'
GPT-5.6 SolGPT-5.6 Sol, low effort, in the post's chart44.8%Announcementchart captioned "Agents' Last Exam: Long-horizon agentic workflows across professional domains" in the opening section (score against API cost in USD; no value labels): point 'GPT-5.6 Sol', effort level 'Low', read from the point's aria-label (load the page in a browser, scroll to the chart and hover or inspect the SVG point)
GPT-5.6 SolGPT-5.6 Sol, medium effort, in the post's chart51.9%Announcementchart captioned "Agents' Last Exam: Long-horizon agentic workflows across professional domains" in the opening section (score against API cost in USD; no value labels): point 'GPT-5.6 Sol', effort level 'Medium', read from the point's aria-label (load the page in a browser, scroll to the chart and hover or inspect the SVG point)
GPT-5.6 SolGPT-5.6 Sol, high effort, in the post's chart52.1%Announcementchart captioned "Agents' Last Exam: Long-horizon agentic workflows across professional domains" in the opening section (score against API cost in USD; no value labels): point 'GPT-5.6 Sol', effort level 'High', read from the point's aria-label (load the page in a browser, scroll to the chart and hover or inspect the SVG point)
GPT-5.6 SolGPT-5.6 Sol, low effort, in OpenAI's later posts45.1%GPT-6 Astra: A new generation of intelligencechart under the heading "Agents' Last Exam" near the top (accuracy against API cost, the default 'API Cost' view; no value labels): point 'GPT-5.6 Sol', reasoning effort 'Low', read from the point's aria-label (load the page in a browser, scroll to the chart and hover or inspect the SVG point)
GPT-5.6 SolGPT-5.6 Sol, medium effort, in OpenAI's later posts52.1%GPT-6 Astra: A new generation of intelligencechart under the heading "Agents' Last Exam" near the top (accuracy against API cost, the default 'API Cost' view; no value labels): point 'GPT-5.6 Sol', reasoning effort 'Medium', read from the point's aria-label (load the page in a browser, scroll to the chart and hover or inspect the SVG point)
GPT-5.6 SolGPT-5.6 Sol, high effort, in OpenAI's later posts52.4%GPT-6 Astra: A new generation of intelligencechart under the heading "Agents' Last Exam" near the top (accuracy against API cost, the default 'API Cost' view; no value labels): point 'GPT-5.6 Sol', reasoning effort 'High', read from the point's aria-label (load the page in a browser, scroll to the chart and hover or inspect the SVG point)
GPT-5.6 SolGPT-5.6 Sol, max effort, in OpenAI's later posts52.8%Introducing GPT-6 Sol and Lunachart under the heading "Agents' Last Exam" (score against cost per task, log scale; no value labels; the second cost-score chart on the page, after AutomationBench): point 'GPT-5.6 Sol', effort 'max', read from the point's aria-label (load the page in a browser, scroll to the chart and hover or inspect the SVG point)

Agents' Last Exam v1 leaderboard, Overall split

ModelValueSources
GPT-5.6GPT-5.6 Sol, xhigh effort, Score53.6%agents-last-exam.orgv1.0 leaderboard, split 'Overall', row 'Codex, gpt-5-6-sol', effort 'XHigh (Best)', column 'Score'
GPT-5.6GPT-5.6 Sol, xhigh effort, Pass Rate30.6%agents-last-exam.orgv1.0 leaderboard, split 'Overall', row 'Codex, gpt-5-6-sol', effort 'XHigh (Best)', column 'Pass Rate'
GPT-5.6GPT-5.6 Sol, low effort, Score45.1%agents-last-exam.orgv1.0 leaderboard, split 'Overall', row 'Codex, gpt-5-6-sol', effort 'Low', column 'Score'
GPT-5.6GPT-5.6 Sol, medium effort, Score52.1%agents-last-exam.orgv1.0 leaderboard, split 'Overall', row 'Codex, gpt-5-6-sol', effort 'Medium', column 'Score'
GPT-5.6GPT-5.6 Sol, high effort, Score52.4%agents-last-exam.orgv1.0 leaderboard, split 'Overall', row 'Codex, gpt-5-6-sol', effort 'High', column 'Score'
GPT-5.6GPT-5.6 Sol, max effort, Score52.8%agents-last-exam.orgv1.0 leaderboard, split 'Overall', row 'Codex, gpt-5-6-sol', effort 'Max', column 'Score'
GPT-5.6GPT-5.6 Sol, max effort, Pass Rate29.6%agents-last-exam.orgv1.0 leaderboard, split 'Overall', row 'Codex, gpt-5-6-sol', effort 'Max', column 'Pass Rate'

Agents' Last Exam v1 leaderboard, ALE-CLI split

ModelValueSources
GPT-5.6GPT-5.6 Sol, xhigh effort, Score53.6%agents-last-exam.orgv1.0 leaderboard, split 'ALE-CLI (linux task only)', row 'Codex, gpt-5-6-sol', effort 'XHigh (Best)', column 'Score'
GPT-5.6GPT-5.6 Sol, xhigh effort, Pass Rate28.6%agents-last-exam.orgv1.0 leaderboard, split 'ALE-CLI (linux task only)', row 'Codex, gpt-5-6-sol', effort 'XHigh (Best)', column 'Pass Rate'
GPT-5.6GPT-5.6 Sol, max effort, Pass Rate26.7%agents-last-exam.orgv1.0 leaderboard, split 'ALE-CLI (linux task only)', row 'Codex, gpt-5-6-sol', effort 'Max', column 'Pass Rate'
GPT-5.6GPT-5.6 Sol, low effort, Score46.8%agents-last-exam.orgv1.0 leaderboard, split 'ALE-CLI (linux task only)', row 'Codex, gpt-5-6-sol', effort 'Low', column 'Score'
GPT-5.6GPT-5.6 Sol, medium effort, Score53.1%agents-last-exam.orgv1.0 leaderboard, split 'ALE-CLI (linux task only)', row 'Codex, gpt-5-6-sol', effort 'Medium', column 'Score'
GPT-5.6GPT-5.6 Sol, high effort, Score51.2%agents-last-exam.orgv1.0 leaderboard, split 'ALE-CLI (linux task only)', row 'Codex, gpt-5-6-sol', effort 'High', column 'Score'
GPT-5.6GPT-5.6 Sol, max effort, Score50.9%agents-last-exam.orgv1.0 leaderboard, split 'ALE-CLI (linux task only)', row 'Codex, gpt-5-6-sol', effort 'Max', column 'Score'

Agents' Last Exam v1, GPT-5.6 Sol in other labs' tables

ModelValueSources
GPT-5.6GPT-5.6 Sol, as Z.ai printed it28.6%GLM-5.3: Frontier Coding with Emergent Cyber Capabilities'Performance across comparison models' table, row 'Agents' Last Exam' / 'ALE-CLI', column 'GPT-5.6 Sol'
GPT-5.6GPT-5.6 Sol, as DeepSeek printed it26.7%DeepSeek-V4.1-Flash: Pushing the Limits of KV Cache CompressionTable 3, p.33, row 'Agents' Last Exam(Pass@1)', column 'GPT-5.6 Sol', 'Max'
GPT-5.6GPT-5.6 Sol, pass, as Alibaba printed it30.6%Qwen3.8-Max: A New Bar for Coding and Cowork'Full Benchmark Table', General Agent block, row 'Agents' Last Exam (Pass / Score)', column 'GPT5.6 Sol (max)', first of the two values
GPT-5.6GPT-5.6 Sol, score, as Alibaba printed it53.6%Qwen3.8-Max: A New Bar for Coding and Cowork'Full Benchmark Table', General Agent block, row 'Agents' Last Exam (Pass / Score)', column 'GPT5.6 Sol (max)', second of the two values

Agents' Last Exam v1, API cost of GPT-5.6 Sol in OpenAI's chart

ModelValueSources
GPT-5.6GPT-5.6 Sol, low effort, API cost in OpenAI's chart248 USDAnnouncementchart captioned "Agents' Last Exam: Long-horizon agentic workflows across professional domains", point 'GPT-5.6 Sol', 'Effort Level: Low', 'API cost (USD)', read from the point's label
GPT-5.6GPT-5.6 Sol, medium effort, API cost in OpenAI's chart519 USDAnnouncementchart captioned "Agents' Last Exam: Long-horizon agentic workflows across professional domains", point 'GPT-5.6 Sol', 'Effort Level: Medium', 'API cost (USD)', read from the point's label
GPT-5.6GPT-5.6 Sol, high effort, API cost in OpenAI's chart577 USDAnnouncementchart captioned "Agents' Last Exam: Long-horizon agentic workflows across professional domains", point 'GPT-5.6 Sol', 'Effort Level: High', 'API cost (USD)', read from the point's label
GPT-5.6GPT-5.6 Sol, xhigh effort, API cost in OpenAI's chart763 USDAnnouncementchart captioned "Agents' Last Exam: Long-horizon agentic workflows across professional domains", point 'GPT-5.6 Sol', 'Effort Level: Xhigh', 'API cost (USD)', read from the point's label
GPT-5.6GPT-5.6 Sol, max effort, API cost in OpenAI's chart1087 USDAnnouncementchart captioned "Agents' Last Exam: Long-horizon agentic workflows across professional domains", point 'GPT-5.6 Sol', 'Effort Level: Max', 'API cost (USD)', read from the point's label

Agents' Last Exam v1 leaderboard, estimated cost of GPT-5.6 Sol

ModelValueSources
GPT-5.6GPT-5.6 Sol, low effort, Overall248 USDagents-last-exam.orgv1.0 leaderboard, split 'Overall', row 'Codex, gpt-5-6-sol', effort 'Low', column 'EST. COST'
GPT-5.6GPT-5.6 Sol, medium effort, Overall519 USDagents-last-exam.orgv1.0 leaderboard, split 'Overall', row 'Codex, gpt-5-6-sol', effort 'Medium', column 'EST. COST'
GPT-5.6GPT-5.6 Sol, high effort, Overall576 USDagents-last-exam.orgv1.0 leaderboard, split 'Overall', row 'Codex, gpt-5-6-sol', effort 'High', column 'EST. COST'
GPT-5.6GPT-5.6 Sol, xhigh effort, Overall772 USDagents-last-exam.orgv1.0 leaderboard, split 'Overall', row 'Codex, gpt-5-6-sol', effort 'Xhigh', column 'EST. COST'
GPT-5.6GPT-5.6 Sol, max effort, Overall1084 USDagents-last-exam.orgv1.0 leaderboard, split 'Overall', row 'Codex, gpt-5-6-sol', effort 'Max', column 'EST. COST'
GPT-5.6GPT-5.6 Sol, low effort, ALE-CLI132 USDagents-last-exam.orgv1.0 leaderboard, split 'ALE-CLI (linux task only)', row 'Codex, gpt-5-6-sol', effort 'Low', column 'EST. COST'
GPT-5.6GPT-5.6 Sol, medium effort, ALE-CLI279 USDagents-last-exam.orgv1.0 leaderboard, split 'ALE-CLI (linux task only)', row 'Codex, gpt-5-6-sol', effort 'Medium', column 'EST. COST'
GPT-5.6GPT-5.6 Sol, high effort, ALE-CLI343 USDagents-last-exam.orgv1.0 leaderboard, split 'ALE-CLI (linux task only)', row 'Codex, gpt-5-6-sol', effort 'High', column 'EST. COST'
GPT-5.6GPT-5.6 Sol, xhigh effort, ALE-CLI446 USDagents-last-exam.orgv1.0 leaderboard, split 'ALE-CLI (linux task only)', row 'Codex, gpt-5-6-sol', effort 'Xhigh', column 'EST. COST'
GPT-5.6GPT-5.6 Sol, max effort, ALE-CLI597 USDagents-last-exam.orgv1.0 leaderboard, split 'ALE-CLI (linux task only)', row 'Codex, gpt-5-6-sol', effort 'Max', column 'EST. COST'

Sources

Models

Related

Tags