CUGA Outperforms Hermes and Deep Agents in a 90-Task AppWorld Comparison

1 reports 1 reporting sources Updated 1 hours ago

Understand the event

AI overview

The CUGA team reported that CUGA completed 81.1% of 90 AppWorld tasks, versus 46.7% for Hermes and 30.0% for Deep Agents, using the same GLM-5.3-flash model. The results suggest the harness affected task completion in this benchmark. The comparison used different setups, including an AppWorld configuration for CUGA that the other harnesses did not use. The team said the 90-task sample may not generalize to other tasks, and the runs did not use policies, so they do not measure policy enforcement.

Generated by AI from reports · Updated 47 minutes ago

Latest development11/3 23:57 (UTC+8)

Same Model, Different Harness: When to Pick CUGA, Hermes, or Deep Agents Hugging Face Blog

Models in this story

GLM-5.3 Flash#28 overall$0.0313 input / $0.0846 output per million tokens

API prices of all models

Report timeline

Follow the reports to explore different sides of the event.

11/3

23:57
Hugging Face BlogPicks
Same Model, Different Harness: When to Pick CUGA, Hermes, or Deep Agents
The CUGA team compares CUGA, Hermes, and Deep Agents on 90 AppWorld tasks using the same GLM-5.3-flash model. CUGA completed 81.1% of tasks, compared with 46.7% for Hermes and 30.0% for Deep Agents; the post also describes CUGA’s runtime policy checks and notes that the benchmark is a limited proxy for enterprise automation.

Model releases, price changes and compute developments, with original sources. Free, confirmed by email, and easy to unsubscribe. Read a past issue

Event heat trend

There is not enough continuous observation data to plot a trend yet.