Skip to content
K = 0.731

Study Finds Coding Agent Harnesses Affect Costs More Than Success Rates

1 reports 1 reporting sources 2 hours agoNew

Understand the event

AI overview

Liu and Han’s study found that coding agent harnesses often changed costs more than success rates in offline SWE-bench Verified tests, with Claude Code and mini-SWE-agent performing similarly while OpenCode showed a significant performance deficit. The researchers tested five models across the shipped versions of Claude Code, OpenCode and mini-SWE-agent. In a comparison covering 447 tasks for two tested models, Claude Code and mini-SWE-agent finished less than two percentage points apart in success rate. System prompts and tool descriptions accounted for much of the cost difference. Liu and Han recommend reporting repeated runs to distinguish the effects of a harness from measurement noise.

Generated by AI from reports · Updated 1 hours ago

Latest development10/24 17:21 (UTC+8)
Fancy harnesses make coding agents more expensive but not better, study findsThe Decoder

Read next

More changes in today’s briefingKimi K3.1 Pro ranks fourth on Artificial Analysis, approaching Claude Opus 5.5

Report timeline

Follow the reports to explore different sides of the event.

10/24
  1. The Decoder
    Fancy harnesses make coding agents more expensive but not better, study finds

    Liu and Han found that coding agent harnesses often changed costs more than success rates in offline SWE-bench Verified tests. Claude Code and mini-SWE-agent finished less than two points apart on 447 tasks for both tested models, while OpenCode showed a significant deficit. System prompts and tool descriptions drove much of the cost difference; the researchers recommend reporting repeat runs to distinguish harness effects from measurement noise.

Keep up with AI, one email a day

Model releases, price changes and compute developments, with original sources. Free, confirmed by email, and easy to unsubscribe. Read a past issue

Event heat trend

There is not enough continuous observation data to plot a trend yet.