{
  "api_version": "1",
  "updated": "2026-10-30",
  "latest_event": "2026-10-29",
  "rights": "© Frontier Log, all rights reserved.",
  "story": {
    "slug": "open-models-frontiercode-harness",
    "title": "FrontierCode 2.0's Top Open Model Depends on the Harness",
    "description": "Mistral Large 4 leads open models on FrontierCode 2.0 in Mistral's Vibe harness but trails DeepSeek and GLM-5.3 in Artificial Analysis's Stirrup harness.",
    "url": "https://frontierlog.ai/stories/open-models-frontiercode-harness",
    "date": "2026-10-30",
    "image": "https://frontierlog.ai/og/story-open-models-frontiercode-harness.png",
    "event": "2026-10-06-mistral-large-4",
    "tags": [
      "coding",
      "open-weights",
      "agents"
    ],
    "steps": [],
    "tables": [
      {
        "title": "FrontierCode 2.0, Cognition's leaderboard",
        "unit": "%",
        "rows": [
          {
            "event": "2026-09-22-claude-opus-5-5",
            "date": "2026-09-22",
            "provider": "anthropic",
            "value": 45.5,
            "printed": null,
            "source": "https://cognition.com/frontiercode",
            "where": "'FrontierCode 2.0 Leaderboard' table, 'All reasoning levels', row 'Opus 5.5' (max), column 'Score'"
          },
          {
            "event": "2026-09-03-gpt-6-astra",
            "date": "2026-09-03",
            "provider": "openai",
            "value": 42.8,
            "printed": null,
            "source": "https://cognition.com/frontiercode",
            "where": "'FrontierCode 2.0 Leaderboard' table, 'All reasoning levels', row 'GPT-6 Astra' (max), column 'Score'"
          },
          {
            "event": "2026-10-28-gemini-4-carbon",
            "date": "2026-10-28",
            "provider": "google",
            "value": 37.5,
            "printed": null,
            "source": "https://cognition.com/frontiercode",
            "where": "'FrontierCode 2.0 Leaderboard' table, 'All reasoning levels', row 'Gemini 4 Carbon' (high), column 'Score'"
          },
          {
            "event": "2026-10-06-mistral-large-4",
            "date": "2026-10-06",
            "provider": "mistral",
            "value": 19.7,
            "printed": null,
            "source": "https://cognition.com/frontiercode",
            "where": "'FrontierCode 2.0 Leaderboard' table, 'All reasoning levels', row 'Mistral Large 4' (high), column 'Score'"
          },
          {
            "event": "2026-09-10-deepseek-v4-1-flash",
            "date": "2026-09-10",
            "provider": "deepseek",
            "value": 19.3,
            "printed": null,
            "source": "https://cognition.com/frontiercode",
            "where": "'FrontierCode 2.0 Leaderboard' table, 'All reasoning levels', row 'DeepSeek V4.1 Flash' (max), column 'Score'"
          },
          {
            "event": "2026-08-14-glm-5-3",
            "date": "2026-08-14",
            "provider": "zai",
            "value": 17.8,
            "printed": null,
            "source": "https://cognition.com/frontiercode",
            "where": "'FrontierCode 2.0 Leaderboard' table, 'All reasoning levels', row 'GLM 5.3' (max), column 'Score'"
          },
          {
            "event": "2026-09-29-gpt-6-1-sol",
            "date": "2026-09-29",
            "provider": "openai",
            "value": 39.0,
            "printed": null,
            "source": "https://cognition.com/frontiercode",
            "where": "'FrontierCode 2.0 Leaderboard' table, 'All reasoning levels', row 'GPT-6.1 Sol' (max), column 'Score'"
          }
        ]
      },
      {
        "title": "FrontierCode 2.0, run by Artificial Analysis",
        "unit": "%",
        "rows": [
          {
            "event": "2026-09-22-claude-opus-5-5",
            "date": "2026-09-22",
            "provider": "anthropic",
            "value": 46.4,
            "printed": null,
            "source": "https://artificialanalysis.ai/evaluations/frontiercode-2-0",
            "where": "'FrontierCode 2.0: Score' chart, bar 'Claude Opus 5.5 (max with fallback)'"
          },
          {
            "event": "2026-09-03-gpt-6-astra",
            "date": "2026-09-03",
            "provider": "openai",
            "value": 42.4,
            "printed": null,
            "source": "https://artificialanalysis.ai/evaluations/frontiercode-2-0",
            "where": "'FrontierCode 2.0: Score' chart, bar 'GPT-6 Astra (max)'"
          },
          {
            "event": "2026-09-29-gpt-6-1-sol",
            "date": "2026-09-29",
            "provider": "openai",
            "value": 40.6,
            "printed": null,
            "source": "https://artificialanalysis.ai/evaluations/frontiercode-2-0",
            "where": "'FrontierCode 2.0: Score' chart, bar 'GPT-6.1 Sol (max)'"
          },
          {
            "event": "2026-10-28-gemini-4-carbon",
            "date": "2026-10-28",
            "provider": "google",
            "value": 40.6,
            "printed": null,
            "source": "https://artificialanalysis.ai/evaluations/frontiercode-2-0",
            "where": "'FrontierCode 2.0: Score' chart, bar 'Gemini 4 Carbon (high)'"
          },
          {
            "event": "2026-10-06-mistral-large-4",
            "date": "2026-10-06",
            "provider": "mistral",
            "value": 15.0,
            "printed": null,
            "source": "https://artificialanalysis.ai/evaluations/frontiercode-2-0",
            "where": "'FrontierCode 2.0: Score' chart, bar 'Mistral Large 4 (high)'"
          },
          {
            "event": "2026-08-14-glm-5-3",
            "date": "2026-08-14",
            "provider": "zai",
            "value": 21.6,
            "printed": null,
            "source": "https://artificialanalysis.ai/evaluations/frontiercode-2-0",
            "where": "'FrontierCode 2.0: Score' chart, bar 'GLM-5.3 (max)'"
          },
          {
            "event": "2026-10-17-deepseek-v4-1-flash-1017",
            "date": "2026-10-17",
            "provider": "deepseek",
            "value": 28.5,
            "printed": null,
            "source": "https://artificialanalysis.ai/evaluations/frontiercode-2-0",
            "where": "'FrontierCode 2.0: Score' chart, bar 'DeepSeek V4.1 Flash 1017 (max)'"
          }
        ]
      }
    ],
    "cites": [
      {
        "url": "https://mistral.ai/news/mistral-large-4-oss/",
        "by": "Mistral AI",
        "medium": "blog post"
      },
      {
        "url": "https://cognition.com/frontiercode",
        "by": "Cognition",
        "medium": "leaderboard"
      },
      {
        "url": "https://artificialanalysis.ai/evaluations/frontiercode-2-0",
        "by": "Artificial Analysis",
        "medium": "evaluation page"
      },
      {
        "url": "https://github.com/ArtificialAnalysis/Stirrup",
        "by": "Artificial Analysis",
        "medium": "GitHub repository"
      }
    ],
    "sources": [
      "https://mistral.ai/news/mistral-large-4-oss/",
      "https://cognition.com/frontiercode",
      "https://artificialanalysis.ai/evaluations/frontiercode-2-0",
      "https://github.com/ArtificialAnalysis/Stirrup"
    ],
    "body": [
      {
        "figure": "main"
      },
      {
        "p": "Mistral released the weights of Mistral Large 4 on Oct 29 and said that on Cognition's FrontierCode 2.0 benchmark, which grades whether a repository maintainer would merge an agent's change, the model \"leads with a score of 19.7%\" ([Mistral's post](https://mistral.ai/news/mistral-large-4-oss/)). On [Cognition's leaderboard](https://cognition.com/frontiercode), that score comes from Mistral's own agent harness, Mistral Vibe, while the other open models there ran in Cognition's Devin CLI. [Artificial Analysis](https://artificialanalysis.ai/evaluations/frontiercode-2-0), which runs FrontierCode 2.0 in its own harness for every model, prints 15.0% for Large 4, behind DeepSeek V4.1 Flash 1017 at 28.5% and GLM-5.3 at 21.6%."
      },
      {
        "h": "The harness changed two days before the weights"
      },
      {
        "p": "Cognition added Mistral Large 4 (Preview) to the leaderboard on Oct 13. On Oct 27, the day its results moved to FrontierCode 2.0, a 132-task suite, its [changelog](https://cognition.com/frontiercode) says it \"Updated Mistral Large 4 to the Mistral Vibe harness (low, medium, and high reasoning), replacing the Devin CLI preview results.\" The same entry notes that \"Points are shaped by evaluation harness.\" The leaderboard no longer shows the Devin CLI results."
      },
      {
        "h": "Each lab's harness on Cognition's board"
      },
      {
        "p": "Cognition runs closed models in their labs' own harnesses: [Claude Opus 5.5](https://frontierlog.ai/model/claude-opus-5-5) in Claude Code at 45.5%, and [GPT-6 Astra](https://frontierlog.ai/model/gpt-6-astra) at 42.8% and [GPT-6.1 Sol](https://frontierlog.ai/model/gpt-6-1-sol) at 39.0% in Codex. It runs [Gemini 4 Carbon](https://frontierlog.ai/model/gemini-4-carbon) in Devin CLI, at 37.5% on high effort. Among the open models, [Large 4](https://frontierlog.ai/model/mistral-large-4) at 19.7% leads [DeepSeek V4.1 Flash](https://frontierlog.ai/model/deepseek-v4-1-flash) at 19.3% and [GLM-5.3](https://frontierlog.ai/model/glm-5-3) at 17.8%, both in Devin CLI."
      },
      {
        "h": "One harness for every model"
      },
      {
        "p": "Artificial Analysis says it runs \"every task three times with our Stirrup agent harness, capped at 500 turns\", with GPT-6 Luna (medium) as the judge ([its FrontierCode 2.0 page](https://artificialanalysis.ai/evaluations/frontiercode-2-0)). [Stirrup](https://github.com/ArtificialAnalysis/Stirrup) is its open-source harness. In that run Opus 5.5 scores 46.4% and GPT-6 Astra 42.4%, and GPT-6.1 Sol and Gemini 4 Carbon both print 40.6%, with Sol listed first. Artificial Analysis ran the Oct 17 update of DeepSeek's model, [V4.1 Flash 1017](https://frontierlog.ai/model/deepseek-v4-1-flash-1017), not the release on Cognition's board."
      },
      {
        "p": "This story quotes Cognition's and Artificial Analysis's published figures directly, an exception to Frontier Log's rule of recording only the labs' own reports, and makes no claim about which ranking is right. Each model is shown at the same reasoning effort in both columns. Earlier stories in this series cover [Terminal-Bench 4.0 by who ran it](https://frontierlog.ai/stories/who-leads-terminal-bench-4), [Gemini 4 Carbon's launch table](https://frontierlog.ai/stories/gemini-4-carbon-aa-scores) and [harness versions](https://frontierlog.ai/stories/does-harness-version-change-score)."
      }
    ],
    "models": [
      "mistral-large-4",
      "deepseek-v4-1-flash",
      "deepseek-v4-1-flash-1017",
      "glm-5-3",
      "claude-opus-5-5",
      "gpt-6-astra",
      "gpt-6-1-sol",
      "gemini-4-carbon"
    ],
    "stories": [
      "who-leads-terminal-bench-4",
      "gemini-4-carbon-aa-scores",
      "does-harness-version-change-score"
    ]
  }
}