Frontier Log

How Google and Thinking Machines Report HLE-Diamond

Gemini 4 Hydrogen and Inkling 2 Small report HLE-Diamond scores that Artificial Analysis ran without tools, the setting of the benchmark creators' main results.

A table comparing the benchmark creator's default settings for HLE-Diamond (CAIS and Scale AI) with the setups behind Gemini 4 Hydrogen's 58.8% (Google, run by Artificial Analysis) and Inkling 2 Small's 28.7% (Thinking Machines, run by Artificial Analysis). Questions: the creator's default is 1,000 (500 reasoning, 500 knowledge), and both labs' scores use all 1,000, the same as the default. Tools: the default is no tools, and both ran without tools, the same. Reasoning effort: the default is high for the main results, with each model's highest setting in an alternate "See reasoning max" view. Gemini 4 Hydrogen's is not stated. Inkling 2 Small's is xhigh (effort 0.99), not comparable with the default. Judge: the HLE-Diamond announcement doesn't state one, and both use GPT-5.6 Luna (medium). Attempts: the creator doesn't state them, and both use pass@1. Token limit: the creator doesn't state one, Gemini 4 Hydrogen's is not stated, and Inkling 2 Small's is a 1M-token trajectory.
Google's Gemini 4 Hydrogen scored 58.8% and Thinking Machines' Inkling 2 Small scored 28.7% on HLE-Diamond, both in Artificial Analysis's runs. Both runs used all 1,000 questions without tools, the benchmark creator's default. Thinking Machines states effort 0.99 (xhigh) for its model. Google doesn't state an effort for this score. Download
PartHLE-Diamond default settings (CAIS and Scale AI)Gemini 4 Hydrogen (Google, run by Artificial Analysis)Inkling 2 Small (Thinking Machines, run by Artificial Analysis)
Questions1,000 (500 reasoning, 500 knowledge)All 1,000All 1,000
ToolsNo toolsNo toolsNo tools
Reasoning effortHigh (main results). Each model's highest setting.Not statedxhigh (effort 0.99)
Judge-GPT-5.6 Luna (medium)GPT-5.6 Luna (medium)
Attempts-pass@1pass@1
Token limit-Not stated1M-token trajectory

HLE-Diamond is a 1,000-question subset of Humanity's Last Exam that the Center for AI Safety and Scale AI released on Sep 22, 2026. The two groups compare models without tools and at reasoning high in their main results, their announcement says. Two labs have since put HLE-Diamond scores in their launch posts. Google's Gemini 4 Hydrogen scored 58.8% and Thinking Machines' Inkling 2 Small scored 28.7% on HLE-Diamond.

Both scores come from Artificial Analysis

Neither lab ran the benchmark itself. Google's evaluation document for Gemini 4 Hydrogen says that "HLE Diamond results are reported from the Artificial Analysis leaderboard." Google announced the model on Nov 12 in its launch post. Thinking Machines' Inkling 2 Small post, published on Nov 5, lists Artificial Analysis as the source of its HLE-Diamond scores.

The same questions and tools as the creators' default

Artificial Analysis says in its methodology that "We evaluate all 1,000 questions without tools." That matches the setting of the creators' main results. The creators also chart scores with web search and code, a setting they define in their evaluation notes.

Artificial Analysis uses GPT-5.6 Luna (medium) as its judge. Its scoring is pass@1, and it grades each answer with "the official HLE judge prompt and response format". The creators' announcement doesn't name a judge model or a number of attempts. The creators' original HLE evaluation code judges with o3-mini by default. Neither the announcement nor that code names the judge of the HLE-Diamond charts.

The two posts state effort differently

The creators chart every model at reasoning high for their main results. A "See reasoning max" toggle on the same charts shows each model at its highest available setting. Thinking Machines states that "All evals run at effort 0.99 (xhigh)" for Inkling 2 Small. Its post also sets "a 1M-token maximum trajectory limit" for every eval. The post gives effort as a number and names the level xhigh. The creators name reasoning levels and don't give effort as a number. Artificial Analysis's methodology doesn't state the effort of its Inkling 2 Small run.

Google's evaluation document says that Google ran its results at the highest thinking settings. Its note on HLE-Diamond says only that the result comes from the Artificial Analysis leaderboard. It doesn't state the thinking setting behind the 58.8%. Artificial Analysis's methodology doesn't list an effort level for each model. Google's launch post lists four thinking levels for the model: low, medium, high (the default) and xhigh. It cites Artificial Analysis's Intelligence Index at both high and xhigh.

Neither model is on the creators' chart

The main chart in the creators' announcement shows ten models, among them Gemini 4 Carbon and Gemini 4 Argon. None of the announcement's charts lists Gemini 4 Hydrogen or Inkling 2 Small.

HLE-Diamond, no tools

ModelScoreSources
Inkling 2 Small28.7%Announcementsection 'Benchmarking Inkling 2 Small', main benchmark table, STEM Reasoning group, row 'HLE-Diamond', column 'Inkling 2 Small' (first model column); runner: list of externally reported evals right after the main benchmark table, before 'Limitations'; tools: the runner's methodology, https://artificialanalysis.ai/methodology/intelligence-benchmarking, section 'HLE-Diamond'
Gemini 4 Hydrogen58.8%Announcementbenchmark table (image), section 'Reasoning', row 'HLE-Diamond', column 'Gemini 4 Hydrogen'; runner: evaluation methodology linked from the post's model card (storage.googleapis.com/deepmind-media/gemini/gemini_4_hydrogen_model_evaluation.pdf), section 'Reasoning'; tools: the runner's methodology, https://artificialanalysis.ai/methodology/intelligence-benchmarking, section 'HLE-Diamond'

Sources

Models

Related

Tags