🔍 Read the full analysis: CUGA, Hermes, And Deep Agents: How To Pick The Right Harness on ThorstenMeyerAI.com
Get the latest gadgets delivered free with Prime
- Fast, free delivery on millions of items
- Prime Video, Amazon Music and more included
- Member-only deals all year
TL;DR
In an IBM-reported AppWorld experiment, CUGA completed 73 of 90 tasks with GLM-5.3-flash, compared with 42 for Hermes and 27 for Deep Agents. The test suggests that an agent’s software harness can affect results on a particular workload, but it does not establish a general ranking or test policy enforcement.
IBM reports, in the original analysis, that its CUGA agent harness completed 73 of 90 tasks in an AppWorld test, while Hermes completed 42 and Deep Agents completed 27, with all three using the same open-weight model, GLM-5.3-flash. The comparison suggests that the software coordinating an AI agent can shape its performance on a defined workload, but it does not establish which harness is best overall.
The experiment used the Harbor evaluation framework and gave each harness one attempt at each of the same 90 tasks across 30 scenarios, according to IBM. The reported task completion rates were 81.1% for CUGA, 46.7% for Hermes and 30.0% for Deep Agents. IBM also reported a scenario measure that counted a scenario only when all three of its variants passed: CUGA passed 20 of 30 scenarios, Hermes eight and Deep Agents two.
The harnesses were not configured identically. CUGA used task decomposition, app-focused tools, verification before execution and automatic reflection. Hermes ran with Harbor’s default setup and AppWorld tools over MCP. Deep Agents used AppWorld tools over MCP with LangChain middleware for tool selection, context summaries, a to-do list, retries and call limits. IBM says it made two fixes to Deep Agents so malformed model replies would not end runs.
IBM says the experiment setup is publicly available through harbor-eval, its Harbor fork. The reported results concern task completion in this specific test; they do not measure cost, latency, production reliability or whether an agent follows organizational policies.
Harness Design Shifts Task Results
The result matters because an agent’s score reflects more than the underlying model. A harness determines how tasks are divided, which tools are offered, and whether actions or outputs are checked. Those choices can influence whether a system completes a sequence of steps, even when the model is held constant.
For teams evaluating agents, the comparison is a reason to test the model, harness and workload together, rather than treating a model benchmark as a complete forecast of performance. The wide spread in this test is a useful signal for workflows resembling AppWorld’s multi-step app tasks. It is not evidence that CUGA would lead on personal assistance, research or other work that Hermes and Deep Agents are designed to support.
The scores also say nothing about whether safeguards work. IBM describes CUGA as offering ways to apply rules through its harness, but policy enforcement was not tested in this experiment. Organizations would need separate evidence before relying on any of the systems for sensitive actions.
Products Worth Considering
Bose QuietComfort Ultra Headphones (2nd Gen), Wireless Bluetooth Noise-Cancelling Over-Ear Headphones, Black
Logitech MX Mechanical Wireless Illuminated Keyboard, Tactile Quiet, Graphite
Apple Watch Series 12 GPS 46mm Smartwatch
As an affiliate, we earn on qualifying purchases.
Three Harnesses, Different Use Cases
The source describes CUGA—short for ConfigUrable Generalist Agent—as IBM’s open-source harness for enterprise work across systems such as customer relationship management and payments. It presents Hermes from Nous Research as aimed at always-on personal assistants, and LangChain’s Deep Agents as suited to long, multi-step work such as research. That makes the comparison a question of fit as well as scores, rather than a universal contest.
AppWorld evaluates multi-step tasks across nine apps and 457 APIs, including email, payments and file storage. IBM characterizes the benchmark as a proxy for some capabilities relevant to enterprise automation, such as reading API documentation, paging through results and changing state across apps. However, the benchmark uses consumer apps; it does not directly show how the harnesses would perform on real enterprise systems.
The source also points to directories tracking nearly 200 agent harnesses and cites a separate comparison in which changing the harness reportedly raised GLM-5.2’s SWE-bench Pro score from 23% to 52%. That is a different evaluation, not evidence from the AppWorld test, and should not be combined with its results.
Limits of the AppWorld Evidence
The comparison covers one model, 90 tasks and one attempt per task. The supplied material does not establish how scores would change with other models, different task sets, repeated runs or production workloads. Because each harness used a different configuration, the test also does not isolate every implementation difference or identify which individual feature drove the results.
It remains unclear how the systems compare on cost, latency, reliability at scale or deployment outcomes. AppWorld’s consumer-app setting does not establish performance on enterprise systems. The experiment also did not test policies, access controls or approval procedures, so it cannot show whether the agents would enforce those requirements in operation. The supplied material provides no independent replication or timeline for a broader comparative study.
Test Against Real Workflows
The reported setup is available through IBM’s harbor-eval fork, giving other teams a way to inspect the experiment. For organizations weighing these systems, the next useful step is to run evaluations on representative workflows using the models and tool configurations they expect to deploy. Repeated runs and varied tasks would help distinguish consistent performance from results limited to a particular sample.
Teams considering enterprise use would also need to test access controls, approval steps and data boundaries in their own environments, along with completion rates, cost and response time. The source material does not state when IBM or another group will publish a broader comparison or independent replication, so those developments remain open.
Key Questions
What did IBM’s AppWorld comparison find?
IBM reported that CUGA completed 73 of 90 tasks, Hermes 42 and Deep Agents 27. All three used GLM-5.3-flash, but their harness configurations differed.
Does the test show CUGA is the best agent harness?
No. It reports results on one benchmark, one model and one set of 90 tasks. It does not establish a general ranking across other tasks or deployment settings.
What does AppWorld test?
AppWorld tests multi-step tasks across nine apps and 457 APIs, including email, payments and file storage. IBM describes it as a proxy for some enterprise-automation capabilities, but the benchmark uses consumer apps.
Did the experiment test policy enforcement?
No. The reported experiment did not use policies, so it provides no evidence about policy enforcement, access controls or approval procedures.
What should organizations do with these results?
Use them as a reason to evaluate the specific model-and-harness combination on representative workflows. Test repeated performance and requirements such as security controls, cost and reliability separately.
Primary source: Hugging Face · via ThorstenMeyerAI.com
