Same Model, Different Harness: When to Pick CUGA, Hermes, or Deep Agents
Same Model, Different Harness: CUGA, Hermes, and Deep Agents — When to Pick CUGA?
- Who
- IBM Research
- What
- compares CUGA, Hermes, and Deep Agents on 90 AppWorld tasks using the same GLM-5.3-flash model
- Quote
We gave three of them the same open-weight model and 90 AppWorld tasks, and changing the harness moved task completion from 30% to 81%.
What happened
The CUGA team compares CUGA, Hermes, and Deep Agents on 90 AppWorld tasks using the same GLM-5.3-flash model. CUGA completed 81.1% of tasks, compared with 46.7% for Hermes and 30.0% for Deep Agents; the post also describes CUGA’s runtime policy checks and notes that the benchmark is a limited proxy for enterprise automation.
Why it matters
The controlled comparison shows how harness choice changes AppWorld completion rates with the model held constant, while the policy section explains how runtime checks differ from prompt instructions.
Read next
See the full storyCUGA Outperforms Hermes and Deep Agents in a 90-Task AppWorld ComparisonThe story so far
The CUGA team reported that CUGA completed 81.1% of 90 AppWorld tasks, versus 46.7% for Hermes and 30.0% for Deep Agents, using the same GLM-5.3-flash model. The results suggest the harness affected task completion in this benchmark. The comparison used different setups, including an AppWorld configuration for CUGA that the other harnesses did not use. The team said the 90-task sample may not generalize to other tasks, and the runs did not use policies, so they do not measure policy enforcement.
Keep up with AI, one email a day
Model releases, price changes and compute developments, with original sources. Free, confirmed by email, and easy to unsubscribe. Read a past issue · 2026-11-03