Does the Harness Version Change a Model's Score?

DeepSeek tested how much the version of an agent harness moves a coding score. In its V4.1 technical report, published with DeepSeek-V4.1-Flash on Sep 10, it ran the model at max effort on four Claude Code versions, from v2.1.105 to v2.1.259. Its Terminal-Bench 2.1 scores stayed between 87.3 and 88.4, and its DeepSWE v1.1 scores between 68.4 and 69.8.

Versions move it a point, harnesses up to nine

The same report ran the model in eight harness setups: Claude Code, Codex, OpenCode, Pi, mini-SWE and three modes of DeepSeek's own harness. There the spread was wider. Terminal-Bench 2.1 ran from 84.1 in Codex to 90.6 in DeepSeek Harness's Minimal mode, and DeepSWE v1.1 from 65.5 in OpenCode to 74.2 in mini-SWE, the model card shows. DeepSeek's headline scores, 90.6 and 74.2, are the highest of the eight setups on each benchmark.

The report gives one version for each of the other harnesses, Codex v0.147.0, OpenCode v1.18.15 and Pi v0.84.2, and none for mini-SWE, a port named mini_swe_v2, or for the Minimal mode of DeepSeek Harness. It says DeepSeek trained the model across several Claude Code versions and other harnesses. An earlier story covered how far other labs' scores move between harnesses.

What other labs name with a score

Some labs name the harness version with a score. MiniMax gave Codex 0.149.1 for the Terminal-Bench 4.0 score of M3.1 Flash on Oct 19, and five Claude Code versions, from 2.1.220 to 2.1.285, for its other benchmarks. Z.ai gave Claude Code 2.1.207 for GLM-5.3-Flash on Aug 26.

Others name the harness alone. Anthropic's Haiku 5.5 system card says its Terminal-Bench 4.0 scores use Claude Code in --bare mode, with no version. DeepSeek's own V4.1-Flash-1017 post on Oct 17 names Claude Code for its Terminal-Bench 4.0 score, also without a version, and OpenAI's GPT-6.1 Terra post on Oct 26 names no harness for its score.