Fancy harnesses make coding agents more expensive but not better, study finds

What happened

Liu and Han found that coding agent harnesses often changed costs more than success rates in offline SWE-bench Verified tests. Claude Code and mini-SWE-agent finished less than two points apart on 447 tasks for both tested models, while OpenCode showed a significant deficit. System prompts and tool descriptions drove much of the cost difference; the researchers recommend reporting repeat runs to distinguish harness effects from measurement noise.

Read next

• More changes in today’s briefingKimi K3.1 Pro ranks fourth on Artificial Analysis, approaching Claude Opus 5.5

The story so far

Liu and Han’s study found that coding agent harnesses often changed costs more than success rates in offline SWE-bench Verified tests, with Claude Code and mini-SWE-agent performing similarly while OpenCode showed a significant performance deficit. The researchers tested five models across the shipped versions of Claude Code, OpenCode and mini-SWE-agent. In a comparison covering 447 tasks for two tested models, Claude Code and mini-SWE-agent finished less than two percentage points apart in success rate. System prompts and tool descriptions accounted for much of the cost difference. Liu and Han recommend reporting repeated runs to distinguish the effects of a harness from measurement noise.