{
  "api_version": "1",
  "updated": "2026-10-18",
  "latest_event": "2026-10-17",
  "rights": "© Frontier Log, all rights reserved.",
  "benchmark": {
    "slug": "aime-2024",
    "name": "AIME 2024",
    "family": "aime",
    "unit": "%",
    "higher_is_better": true,
    "status": "superseded",
    "benchmark_url": "https://maa.org/amc/aime",
    "introduced": "2024-02-07",
    "creator": "Mathematical Association of America",
    "introduced_source": "https://maa.org/sites/default/files/Programs/2024%20AIME%20TM.pdf",
    "settings": [
      {
        "setting": "no-tools",
        "name": "No tools",
        "url": "https://frontierlog.ai/benchmark/aime-2024",
        "results": 7
      }
    ]
  },
  "total": 7,
  "results": [
    {
      "id": "deepseek-r1:aime-2024:no-tools:1",
      "model": "deepseek-r1",
      "model_name": "DeepSeek-R1",
      "model_date": "2025-01-20",
      "event": "https://frontierlog.ai/event/2025-01-20-deepseek-r1",
      "benchmark": "aime-2024",
      "benchmark_name": "AIME 2024",
      "setting": "no-tools",
      "setting_name": "No tools",
      "score": 79.8,
      "unit": "%",
      "printed": "79.8",
      "reported_by": "deepseek",
      "label": "DeepSeek-R1",
      "source": "https://api-docs.deepseek.com/news/news250120/",
      "source_title": "DeepSeek-R1 Release",
      "source_date": "2025-01-20",
      "where": "bar chart image (first image), group 'AIME 2024 (Pass@1)', first bar (legend 'DeepSeek-R1'), value label",
      "own": true,
      "primary": true,
      "parallel": false,
      "notes": "Pass@1; post states no tool use either way, so no-tools by default. The distill table image prints AIME 2024 pass@1/cons@64 only for the distilled models and competitors, not for DeepSeek-R1.",
      "caveat": "",
      "human_verified": true,
      "human_verified_on": "2026-10-18",
      "source_image": "https://frontierlog.ai/sources/9ae29b740936681e.png",
      "source_image_original": "https://cdn.deepseek.com/api-docs/deepseek_r1_benchmark.png"
    },
    {
      "id": "claude-3-7-sonnet:aime-2024:no-tools:1",
      "model": "claude-3-7-sonnet",
      "model_name": "Claude 3.7 Sonnet",
      "model_date": "2025-02-24",
      "event": "https://frontierlog.ai/event/2025-02-24-claude-3-7-sonnet",
      "benchmark": "aime-2024",
      "benchmark_name": "AIME 2024",
      "setting": "no-tools",
      "setting_name": "No tools",
      "score": 61.3,
      "unit": "%",
      "printed": "61.3%",
      "reported_by": "anthropic",
      "label": "Claude 3.7 Sonnet (64K extended thinking)",
      "source": "https://www.anthropic.com/news/claude-3-7-sonnet",
      "source_title": "Claude 3.7 Sonnet and Claude Code",
      "source_date": "2025-02-24",
      "where": "benchmark table image, row 'High school math competition / AIME 2024', column 'Claude 3.7 Sonnet 64K extended thinking', first number of the cell",
      "own": true,
      "primary": true,
      "parallel": false,
      "notes": "64K extended thinking; pass@1 averaged over up to 16 trials. Chosen as primary over the no-extended-thinking column (both are own, non-parallel); flag if the other should be primary.",
      "caveat": "",
      "human_verified": true,
      "human_verified_on": "2026-10-18",
      "source_image": "https://frontierlog.ai/sources/7018392ca041e885.png",
      "source_image_original": "https://www-cdn.anthropic.com/images/4zrzovbb/website/654cf6680d32858dfba9af644f8c4a5b04425af1-2600x2360.png"
    },
    {
      "id": "claude-3-7-sonnet:aime-2024:no-tools:2",
      "model": "claude-3-7-sonnet",
      "model_name": "Claude 3.7 Sonnet",
      "model_date": "2025-02-24",
      "event": "https://frontierlog.ai/event/2025-02-24-claude-3-7-sonnet",
      "benchmark": "aime-2024",
      "benchmark_name": "AIME 2024",
      "setting": "no-tools",
      "setting_name": "No tools",
      "score": 80.0,
      "unit": "%",
      "printed": "80.0%",
      "reported_by": "anthropic",
      "label": "Claude 3.7 Sonnet (64K extended thinking)",
      "source": "https://www.anthropic.com/news/claude-3-7-sonnet",
      "source_title": "Claude 3.7 Sonnet and Claude Code",
      "source_date": "2025-02-24",
      "where": "benchmark table image, row 'AIME 2024', column 'Claude 3.7 Sonnet 64K extended thinking', second number of the cell (after the slash)",
      "own": true,
      "primary": false,
      "parallel": true,
      "notes": "64K extended thinking. Footnote 3: high score uses internal scoring with parallel test-time compute. Methodology: pass@1 averaged over up to 16 trials for AIME.",
      "caveat": "",
      "human_verified": true,
      "human_verified_on": "2026-10-18",
      "source_image": "https://frontierlog.ai/sources/7018392ca041e885.png",
      "source_image_original": "https://www-cdn.anthropic.com/images/4zrzovbb/website/654cf6680d32858dfba9af644f8c4a5b04425af1-2600x2360.png"
    },
    {
      "id": "claude-3-7-sonnet:aime-2024:no-tools:3",
      "model": "claude-3-7-sonnet",
      "model_name": "Claude 3.7 Sonnet",
      "model_date": "2025-02-24",
      "event": "https://frontierlog.ai/event/2025-02-24-claude-3-7-sonnet",
      "benchmark": "aime-2024",
      "benchmark_name": "AIME 2024",
      "setting": "no-tools",
      "setting_name": "No tools",
      "score": 23.3,
      "unit": "%",
      "printed": "23.3%",
      "reported_by": "anthropic",
      "label": "Claude 3.7 Sonnet (No extended thinking)",
      "source": "https://www.anthropic.com/news/claude-3-7-sonnet",
      "source_title": "Claude 3.7 Sonnet and Claude Code",
      "source_date": "2025-02-24",
      "where": "benchmark table image, row 'AIME 2024', column 'Claude 3.7 Sonnet No extended thinking'",
      "own": true,
      "primary": false,
      "parallel": false,
      "notes": "No extended thinking; pass@1 averaged over up to 16 trials.",
      "caveat": "",
      "human_verified": true,
      "human_verified_on": "2026-10-18",
      "source_image": "https://frontierlog.ai/sources/7018392ca041e885.png",
      "source_image_original": "https://www-cdn.anthropic.com/images/4zrzovbb/website/654cf6680d32858dfba9af644f8c4a5b04425af1-2600x2360.png"
    },
    {
      "id": "gpt-4-5:aime-2024:no-tools:1",
      "model": "gpt-4-5",
      "model_name": "GPT-4.5",
      "model_date": "2025-02-27",
      "event": "https://frontierlog.ai/event/2025-02-27-gpt-4-5",
      "benchmark": "aime-2024",
      "benchmark_name": "AIME 2024",
      "setting": "no-tools",
      "setting_name": "No tools",
      "score": 36.7,
      "unit": "%",
      "printed": "36.7%",
      "reported_by": "openai",
      "label": "GPT‑4.5",
      "source": "https://openai.com/index/introducing-gpt-4-5",
      "source_title": "Introducing GPT-4.5",
      "source_date": "2025-02-27",
      "where": "main benchmark table (HTML), row 'AIME '24 (math)', column 'GPT-4.5'",
      "own": true,
      "primary": true,
      "parallel": false,
      "notes": "HTML table; the post prints no tools setting for this row (GPT-4.5 is a non-reasoning model and the table has no tool notes; the '*' footnote marks only the SWE rows), so no-tools is assumed. The fetched page text was truncated, so footnotes without a % sign are not visible in the extract.",
      "caveat": "",
      "human_verified": true,
      "human_verified_on": "2026-10-18"
    },
    {
      "id": "gemini-2-5-pro:aime-2024:no-tools:1",
      "model": "gemini-2-5-pro",
      "model_name": "Gemini 2.5 Pro",
      "model_date": "2025-03-25",
      "event": "https://frontierlog.ai/event/2025-03-25-gemini-2-5-pro",
      "benchmark": "aime-2024",
      "benchmark_name": "AIME 2024",
      "setting": "no-tools",
      "setting_name": "No tools",
      "score": 92.0,
      "unit": "%",
      "printed": "92.0%",
      "reported_by": "google",
      "label": "Gemini 2.5 Pro Experimental (03-25)",
      "source": "https://blog.google/innovation-and-ai/models-and-research/google-deepmind/gemini-model-thinking-updates-march-2025/",
      "source_title": "Gemini 2.5: Our most intelligent AI model",
      "source_date": "2025-03-25",
      "where": "benchmark table image (gemini_benchmarks_cropped GIF), row 'Mathematics AIME 2024', sub-row 'single attempt (pass@1)', column 'Gemini 2.5 Pro Experimental (03-25)'",
      "own": true,
      "primary": true,
      "parallel": false,
      "notes": "Methodology under the table: all Gemini 2.5 Pro scores are pass@1 (no majority voting or parallel test-time compute), run via AI Studio API model gemini-2.5-pro-exp-03-25 with default sampling, averaged over multiple trials for smaller benchmarks. No tools printed. The 'multiple attempts' sub-row is a dash for Gemini. The hero chart image does not show AIME 2024.",
      "caveat": "",
      "human_verified": true,
      "human_verified_on": "2026-10-18",
      "source_image": "https://frontierlog.ai/sources/a8f4323681d1a15d.gif",
      "source_image_original": "https://storage.googleapis.com/gweb-uniblog-publish-prod/original_images/gemini_benchmarks_cropped_light2x_1PPmDuP.gif"
    },
    {
      "id": "openai-o3:aime-2024:no-tools:1",
      "model": "openai-o3",
      "model_name": "OpenAI o3",
      "model_date": "2025-04-16",
      "event": "https://frontierlog.ai/event/2025-04-16-o3-and-o4-mini",
      "benchmark": "aime-2024",
      "benchmark_name": "AIME 2024",
      "setting": "no-tools",
      "setting_name": "No tools",
      "score": 91.6,
      "unit": "%",
      "printed": "91.6",
      "reported_by": "openai",
      "label": "o3 (no tools)",
      "source": "https://openai.com/index/introducing-o3-and-o4-mini",
      "source_title": "Introducing OpenAI o3 and o4-mini",
      "source_date": "2025-04-16",
      "where": "'What's changed' section, bar chart 'AIME 2024 / Competition Math', bar labelled 'o3 (no tools)' (3rd bar)",
      "own": true,
      "primary": true,
      "parallel": false,
      "notes": "Read from the full-page screenshot (shots.jsonl part 2) of the HTML bar chart; value label above the bar. Post note under the chart set: 'All models are evaluated at high reasoning effort settings, similar to variants like o4-mini-high in ChatGPT.' Printed without % (y-axis 'Accuracy (%)'). Read from a full-page screenshot of the post (para-kb image /api/images/79), as openai.com is bot-walled.",
      "caveat": "",
      "human_verified": true,
      "human_verified_on": "2026-10-18"
    }
  ]
}