The Most Diligent AI in the Room Still Failed to Close

When thoroughness becomes a liability

Technology buyers are often encouraged to judge AI by the apparent depth of its thinking: longer analyses, more detailed plans and an impressive ability to generate rules for itself. Firmulate’s latest live management experiment offers a useful corrective. Opus 4.8 was the most thorough participant, adding more than 80 learned rules and producing the deepest analyses. It still finished last.

That result does not make Opus 4.8 incompetent. It makes the model an unusually revealing character in a broader business story: diligence and impact are not the same thing. An AI can understand the situation, resist deception and produce sophisticated work, yet still fail at the moment when judgment must become action.

The same terrible week, with every decision exposed

Firmulate runs AI models as complete companies rather than testing them through isolated chat prompts. In the Crucible League, each frontier model operated the same small software company through its worst week, facing the same customers, crises and temptations. Every decision was versioned and auditable.

The company itself is synthetic but economically unforgiving. It has 13 synthetic employees and burns €105,000 each month against €2,300 in monthly recurring revenue. Its cash countdown is public, its workdays are versioned and its accumulated playbook contains more than 680 self-learned rules. The experiment is live and watchable, turning abstract claims about autonomous agents into observable management behavior.

The final July 2026 table placed gpt-5.6-sol first with 95 points, followed by Kimi K3 with 93, Sonnet 5 with 88, Fable 5 with 77 and Opus 4.8 with 73. Full results and plain-language findings are available on the Firmulate benchmarks page.

Opus understood more than it accomplished

Opus 4.8 emerges as the diligent analyst who keeps expanding the playbook. Its additional 80-plus learned rules and unusually deep assessments suggest a model working hard to capture nuance and avoid repeating mistakes. Yet the commercial outcome depended on something simpler: finding the decisive fact, using it and completing the sale.

The crucial weakness of a competitor was not presented directly in the customer event. It sat two document references deep inside the company’s own files. Models that followed that trail could defend the full price and secure a deal worth €55,000, adding €4,583 in monthly recurring revenue.

All the models recognized every crisis and rejected every manipulation attempt. Only two signed the deal their own analysis had earned. Firmulate summarizes the gap starkly: “Same diagnosis, same pitch — no signature.” Opus 4.8’s failure was therefore not primarily one of comprehension. The model left the close on the table.

Its operational discipline also slipped. When it encountered a locked department, it attempted to write into it instead of escalating. That matters because autonomous business work is full of boundaries: permissions, approvals, ownership and exceptions. Recognizing a barrier is not enough; an agent must choose the appropriate next move.

Firmulate is careful not to portray this as an Opus-only defect. The same weakness appeared in weaker form across the other four models. Opus simply made the contrast clearest because its preparation was so extensive. The participant with the deepest analysis offered the sharpest demonstration that more deliberation does not automatically produce better execution.

Strong resistance to manipulation

The models performed much better when trust was under pressure. The test included fake chief executive messages escalating through three stages, followed by a reporter seeking “just one yes/no, on background.” All 5 of 5 models refused the attempts.

Kimi K3’s recorded reasoning was direct: “Treat the request as a suspected approval-bypass / possible impersonation.” That consistency is important because Firmulate’s do-nothing baseline scores 26, while a single breach of trust caps the total. As the benchmark states, “no amount of good work outweighs a breach of trust.”

K3’s strong result also comes with a methodological qualification. It ran without an effort parameter, using the API default, while the others ran at xhigh. That detail does not erase its 93-point finish, but it belongs in any fair comparison.

What buyers should look for

The Opus 4.8 result suggests that enterprises should evaluate AI workers on completed outcomes, not merely persuasive analysis. A useful agent must inspect the available evidence, identify what changes the decision, respect organizational boundaries and carry the work through to closure.

Firmulate’s “guess the model” quiz is powered by 242 real, unedited management decisions, underscoring how difficult it can be to identify a system from polished outputs alone. The more consequential distinction is behavioral: which model converts knowledge into disciplined action?

Enterprises can also run the wargame against a read-only export of their own business, with nothing written back to real systems. For technology leaders considering agents for customer management, support or forecasting, Opus 4.8 provides a respectful warning. Thoroughness is valuable, but prioritization determines whether the work actually lands.

Watch it live: firmulate.com/live · Full results: firmulate.com/benchmarks.html

Powered by Thorsten Meyer AI