{
  "api_version": "1",
  "updated": "2026-10-29",
  "latest_event": "2026-10-28",
  "rights": "© Frontier Log, all rights reserved.",
  "story": {
    "slug": "does-harness-version-change-score",
    "title": "Does the Harness Version Change a Model's Score?",
    "description": "DeepSeek ran V4.1-Flash on four Claude Code versions and its scores moved about a point, against up to 8.7 points across agent harnesses.",
    "url": "https://frontierlog.ai/stories/does-harness-version-change-score",
    "date": "2026-10-29",
    "image": "https://frontierlog.ai/og/story-does-harness-version-change-score-versions.png",
    "event": "2026-09-10-deepseek-v4-1-flash",
    "tags": [
      "coding",
      "agents"
    ],
    "images": [
      "https://frontierlog.ai/og/story-does-harness-version-change-score-versions.png"
    ],
    "steps": [],
    "tables": [
      {
        "title": "Terminal-Bench 2.1 by harness and version",
        "unit": "%",
        "rows": [
          {
            "event": "2026-09-10-deepseek-v4-1-flash",
            "date": "2026-09-10",
            "provider": "deepseek",
            "value": 87.3,
            "printed": "87.3",
            "source": "https://huggingface.co/deepseek-ai/DeepSeek-V4.1-Flash/blob/main/DeepSeek_V41_Tech_Report.pdf",
            "where": "technical report, Appendix Table 5 (performance across Claude Code versions), row 'Terminal-Bench v2.1 (Pass@1)', column 'v2.1.105'"
          },
          {
            "event": "2026-09-10-deepseek-v4-1-flash",
            "date": "2026-09-10",
            "provider": "deepseek",
            "value": 88.4,
            "printed": "88.4",
            "source": "https://huggingface.co/deepseek-ai/DeepSeek-V4.1-Flash/blob/main/DeepSeek_V41_Tech_Report.pdf",
            "where": "technical report, Appendix Table 5 (performance across Claude Code versions), row 'Terminal-Bench v2.1 (Pass@1)', column 'v2.1.238'"
          },
          {
            "event": "2026-09-10-deepseek-v4-1-flash",
            "date": "2026-09-10",
            "provider": "deepseek",
            "value": 88,
            "printed": "88.0",
            "source": "https://huggingface.co/deepseek-ai/DeepSeek-V4.1-Flash",
            "where": "model card, 'Performance across agent scaffolds' table, row 'Terminal-Bench 2.1 (Pass@1)', column 'Claude Code'"
          },
          {
            "event": "2026-09-10-deepseek-v4-1-flash",
            "date": "2026-09-10",
            "provider": "deepseek",
            "value": 87.6,
            "printed": "87.6",
            "source": "https://huggingface.co/deepseek-ai/DeepSeek-V4.1-Flash/blob/main/DeepSeek_V41_Tech_Report.pdf",
            "where": "technical report, Appendix Table 5 (performance across Claude Code versions), row 'Terminal-Bench v2.1 (Pass@1)', column 'v2.1.259'"
          },
          {
            "event": "2026-09-10-deepseek-v4-1-flash",
            "date": "2026-09-10",
            "provider": "deepseek",
            "value": 84.1,
            "printed": "84.1",
            "source": "https://huggingface.co/deepseek-ai/DeepSeek-V4.1-Flash",
            "where": "model card, 'Performance across agent scaffolds' table, row 'Terminal-Bench 2.1 (Pass@1)', column 'Codex'"
          },
          {
            "event": "2026-09-10-deepseek-v4-1-flash",
            "date": "2026-09-10",
            "provider": "deepseek",
            "value": 85,
            "printed": "85.0",
            "source": "https://huggingface.co/deepseek-ai/DeepSeek-V4.1-Flash",
            "where": "model card, 'Performance across agent scaffolds' table, row 'Terminal-Bench 2.1 (Pass@1)', column 'OpenCode'"
          },
          {
            "event": "2026-09-10-deepseek-v4-1-flash",
            "date": "2026-09-10",
            "provider": "deepseek",
            "value": 86.1,
            "printed": "86.1",
            "source": "https://huggingface.co/deepseek-ai/DeepSeek-V4.1-Flash",
            "where": "model card, 'Performance across agent scaffolds' table, row 'Terminal-Bench 2.1 (Pass@1)', column 'Pi'"
          },
          {
            "event": "2026-09-10-deepseek-v4-1-flash",
            "date": "2026-09-10",
            "provider": "deepseek",
            "value": 90.3,
            "printed": "90.3",
            "source": "https://huggingface.co/deepseek-ai/DeepSeek-V4.1-Flash",
            "where": "model card, 'Performance across agent scaffolds' table, row 'Terminal-Bench 2.1 (Pass@1)', column 'mini-SWE'"
          },
          {
            "event": "2026-09-10-deepseek-v4-1-flash",
            "date": "2026-09-10",
            "provider": "deepseek",
            "value": 90.6,
            "printed": "90.6",
            "source": "https://api-docs.deepseek.com/news/news260910/",
            "where": "benchmark table image (v4.1_260910_benchmark_table_en), row 'Terminal-Bench 2.1', column 'DeepSeek V4.1-Flash' (highlighted)"
          },
          {
            "event": "2026-09-10-deepseek-v4-1-flash",
            "date": "2026-09-10",
            "provider": "deepseek",
            "value": 85.8,
            "printed": "85.8",
            "source": "https://huggingface.co/deepseek-ai/DeepSeek-V4.1-Flash",
            "where": "model card, 'Performance across agent scaffolds' table, row 'Terminal-Bench 2.1 (Pass@1)', column 'DSH Standard'"
          },
          {
            "event": "2026-09-10-deepseek-v4-1-flash",
            "date": "2026-09-10",
            "provider": "deepseek",
            "value": 85.8,
            "printed": "85.8",
            "source": "https://huggingface.co/deepseek-ai/DeepSeek-V4.1-Flash",
            "where": "model card, 'Performance across agent scaffolds' table, row 'Terminal-Bench 2.1 (Pass@1)', column 'DSH PTC'"
          }
        ]
      },
      {
        "title": "DeepSWE v1.1 by harness and version",
        "unit": "%",
        "rows": [
          {
            "event": "2026-09-10-deepseek-v4-1-flash",
            "date": "2026-09-10",
            "provider": "deepseek",
            "value": 68.4,
            "printed": "68.4",
            "source": "https://huggingface.co/deepseek-ai/DeepSeek-V4.1-Flash/blob/main/DeepSeek_V41_Tech_Report.pdf",
            "where": "technical report, Appendix Table 5 (performance across Claude Code versions), row 'DeepSWE v1.1 (Resolved)', column 'v2.1.105'"
          },
          {
            "event": "2026-09-10-deepseek-v4-1-flash",
            "date": "2026-09-10",
            "provider": "deepseek",
            "value": 68.7,
            "printed": "68.7",
            "source": "https://huggingface.co/deepseek-ai/DeepSeek-V4.1-Flash/blob/main/DeepSeek_V41_Tech_Report.pdf",
            "where": "technical report, Appendix Table 5 (performance across Claude Code versions), row 'DeepSWE v1.1 (Resolved)', column 'v2.1.238'"
          },
          {
            "event": "2026-09-10-deepseek-v4-1-flash",
            "date": "2026-09-10",
            "provider": "deepseek",
            "value": 69.8,
            "printed": "69.8",
            "source": "https://huggingface.co/deepseek-ai/DeepSeek-V4.1-Flash",
            "where": "model card, 'Performance across agent scaffolds' table, row 'DeepSWE v1.1 (Resolved)', column 'Claude Code'"
          },
          {
            "event": "2026-09-10-deepseek-v4-1-flash",
            "date": "2026-09-10",
            "provider": "deepseek",
            "value": 68.6,
            "printed": "68.6",
            "source": "https://huggingface.co/deepseek-ai/DeepSeek-V4.1-Flash/blob/main/DeepSeek_V41_Tech_Report.pdf",
            "where": "technical report, Appendix Table 5 (performance across Claude Code versions), row 'DeepSWE v1.1 (Resolved)', column 'v2.1.259'"
          },
          {
            "event": "2026-09-10-deepseek-v4-1-flash",
            "date": "2026-09-10",
            "provider": "deepseek",
            "value": 65.6,
            "printed": "65.6",
            "source": "https://huggingface.co/deepseek-ai/DeepSeek-V4.1-Flash",
            "where": "model card, 'Performance across agent scaffolds' table, row 'DeepSWE v1.1 (Resolved)', column 'Codex'"
          },
          {
            "event": "2026-09-10-deepseek-v4-1-flash",
            "date": "2026-09-10",
            "provider": "deepseek",
            "value": 65.5,
            "printed": "65.5",
            "source": "https://huggingface.co/deepseek-ai/DeepSeek-V4.1-Flash",
            "where": "model card, 'Performance across agent scaffolds' table, row 'DeepSWE v1.1 (Resolved)', column 'OpenCode'"
          },
          {
            "event": "2026-09-10-deepseek-v4-1-flash",
            "date": "2026-09-10",
            "provider": "deepseek",
            "value": 66.2,
            "printed": "66.2",
            "source": "https://huggingface.co/deepseek-ai/DeepSeek-V4.1-Flash",
            "where": "model card, 'Performance across agent scaffolds' table, row 'DeepSWE v1.1 (Resolved)', column 'Pi'"
          },
          {
            "event": "2026-09-10-deepseek-v4-1-flash",
            "date": "2026-09-10",
            "provider": "deepseek",
            "value": 74.2,
            "printed": "74.2",
            "source": "https://api-docs.deepseek.com/news/news260910/",
            "where": "benchmark table image (second image), row 'DeepSWE v1.1', column 'DeepSeek V4.1-Flash' (the first chart image, v4.1_260910_benchmark.png, prints the same value label)"
          },
          {
            "event": "2026-09-10-deepseek-v4-1-flash",
            "date": "2026-09-10",
            "provider": "deepseek",
            "value": 72.6,
            "printed": "72.6",
            "source": "https://huggingface.co/deepseek-ai/DeepSeek-V4.1-Flash",
            "where": "model card, 'Performance across agent scaffolds' table, row 'DeepSWE v1.1 (Resolved)', column 'DSH Minimal'"
          },
          {
            "event": "2026-09-10-deepseek-v4-1-flash",
            "date": "2026-09-10",
            "provider": "deepseek",
            "value": 70.5,
            "printed": "70.5",
            "source": "https://huggingface.co/deepseek-ai/DeepSeek-V4.1-Flash",
            "where": "model card, 'Performance across agent scaffolds' table, row 'DeepSWE v1.1 (Resolved)', column 'DSH Standard'"
          },
          {
            "event": "2026-09-10-deepseek-v4-1-flash",
            "date": "2026-09-10",
            "provider": "deepseek",
            "value": 67.6,
            "printed": "67.6",
            "source": "https://huggingface.co/deepseek-ai/DeepSeek-V4.1-Flash",
            "where": "model card, 'Performance across agent scaffolds' table, row 'DeepSWE v1.1 (Resolved)', column 'DSH PTC'"
          }
        ]
      }
    ],
    "sources": [
      "https://api-docs.deepseek.com/news/news260910/",
      "https://huggingface.co/deepseek-ai/DeepSeek-V4.1-Flash",
      "https://huggingface.co/deepseek-ai/DeepSeek-V4.1-Flash/blob/main/DeepSeek_V41_Tech_Report.pdf",
      "https://www.minimax.io/blog/minimax-m31f",
      "https://z.ai/blog/glm-5.3-flash",
      "https://www.anthropic.com/claude-haiku-5-5-system-card",
      "https://api-docs.deepseek.com/news/news261017/",
      "https://openai.com/index/introducing-gpt-6-1-terra"
    ],
    "body": [
      {
        "figure": "versions"
      },
      {
        "p": "DeepSeek tested how much the version of an agent harness moves a coding score. In [its V4.1 technical report](https://huggingface.co/deepseek-ai/DeepSeek-V4.1-Flash/blob/main/DeepSeek_V41_Tech_Report.pdf), published with DeepSeek-V4.1-Flash on Sep 10, it ran the model at max effort on four Claude Code versions, from v2.1.105 to v2.1.259. Its Terminal-Bench 2.1 scores stayed between 87.3 and 88.4, and its DeepSWE v1.1 scores between 68.4 and 69.8."
      },
      {
        "h": "Versions move it a point, harnesses up to nine"
      },
      {
        "p": "The same report ran the model in eight harness setups: Claude Code, Codex, OpenCode, Pi, mini-SWE and three modes of DeepSeek's own harness. There the spread was wider. Terminal-Bench 2.1 ran from 84.1 in Codex to 90.6 in DeepSeek Harness's Minimal mode, and DeepSWE v1.1 from 65.5 in OpenCode to 74.2 in mini-SWE, [the model card](https://huggingface.co/deepseek-ai/DeepSeek-V4.1-Flash) shows. DeepSeek's headline scores, 90.6 and 74.2, are the highest of the eight setups on each benchmark."
      },
      {
        "p": "The report gives one version for each of the other harnesses, Codex v0.147.0, OpenCode v1.18.15 and Pi v0.84.2, and none for mini-SWE, a port named mini_swe_v2, or for the Minimal mode of DeepSeek Harness. It says DeepSeek trained the model across several Claude Code versions and other harnesses. [An earlier story](https://frontierlog.ai/stories/the-harness-is-part-of-the-score) covered how far other labs' scores move between harnesses."
      },
      {
        "h": "What other labs name with a score"
      },
      {
        "p": "Some labs name the harness version with a score. MiniMax gave Codex 0.149.1 for the Terminal-Bench 4.0 score of [M3.1 Flash](https://www.minimax.io/blog/minimax-m31f) on Oct 19, and five Claude Code versions, from 2.1.220 to 2.1.285, for its other benchmarks. Z.ai gave Claude Code 2.1.207 for [GLM-5.3-Flash](https://z.ai/blog/glm-5.3-flash) on Aug 26."
      },
      {
        "p": "Others name the harness alone. Anthropic's [Haiku 5.5 system card](https://www.anthropic.com/claude-haiku-5-5-system-card) says its Terminal-Bench 4.0 scores use Claude Code in --bare mode, with no version. DeepSeek's own [V4.1-Flash-1017 post](https://api-docs.deepseek.com/news/news261017/) on Oct 17 names Claude Code for its Terminal-Bench 4.0 score, also without a version, and OpenAI's [GPT-6.1 Terra post](https://openai.com/index/introducing-gpt-6-1-terra) on Oct 26 names no harness for its score."
      }
    ],
    "models": [
      "deepseek-v4-1-flash",
      "deepseek-v4-1-flash-1017",
      "minimax-m3-1-flash",
      "glm-5-3-flash",
      "claude-haiku-5-5",
      "gpt-6-1-terra"
    ],
    "stories": [
      "the-harness-is-part-of-the-score"
    ]
  }
}