🔍 Read the full analysis: Could A Tough Week Expose AI Agent Risks In Your Business? on ThorstenMeyerAI.com
Get the latest gadgets delivered free with Prime
- Fast, free delivery on millions of items
- Prime Video, Amazon Music and more included
- Member-only deals all year
TL;DR
Firmulate says five frontier models recognized every crisis and refused all staged manipulation attempts in a simulated software company’s difficult week. The results diverged on execution: only two signed a €55,000 deal, and the company is offering read-only pilots to test models against businesses’ own data.
Firmulate’s July 2026 Crucible League, described in the original analysis, found that five frontier AI models identified every crisis and refused every staged manipulation attempt in a simulated software company, but only two signed a €55,000 deal that their own analysis supported. The company is now offering enterprise pilots that run similar tests on read-only exports of businesses’ data, aiming to show how agents behave under pressure without writing to live systems.
The final standings were gpt-5.6-sol with 95 points, Kimi K3 with 93, Sonnet 5 with 88, Fable 5 with 77 and Opus 4.8 with 73. A do-nothing baseline scored 26. Firmulate says partial progress counted toward scores, while a breach of trust capped a participant’s total. The standings describe performance in this experiment, not a general ranking of the models across tasks.
The key separation came after the models diagnosed the sales opportunity. Firmulate says the competitive weakness that could support the deal was buried two document references deep in the company files. Models that found and used that information won the deal at full price, adding €4,583 in monthly recurring revenue. The company’s summary is: “Same diagnosis, same pitch — no signature.” The reported outcome suggests that recognizing a problem and making a persuasive case did not always lead to completing the business action.
The test also presented three escalating fake CEO messages and a reporter’s request for a yes-or-no answer “on background.” Firmulate reports that all five models refused. It says every decision was versioned and auditable. Opus 4.8 produced the deepest analyses and added 80 learned rules, but finished last; Firmulate says it left the deal unsigned and tried to write into a locked department rather than escalate. A weaker form of that boundary issue appeared in all four models, according to the company.
From Crisis Detection to Execution
The results focus attention on a gap that matters to companies considering automation: an agent can identify a crisis without completing the appropriate next step. A business may need an agent to find supporting evidence across internal files, act on a justified opportunity and stop or escalate when a system boundary blocks its preferred route. Firmulate’s experiment suggests those behaviors can vary even when participants diagnose the same situation.
The proposed pilot turns that question toward a company’s own operating information. Firmulate says it uses a read-only export to run crisis scenarios and produce a board report with model rankings and weak points in existing playbooks. If the process works as described, companies could inspect how agents handle their customers, pipeline and rules before connecting agents to systems where they can make changes. The published league alone does not establish how a model would perform in a particular company or live environment.
How the Crucible League Worked
Firmulate’s live experiment follows a synthetic software company with 13 employees and simulated financial pressures. The company reports monthly burn of €105,000 against €2,300 in monthly recurring revenue, alongside a public cash countdown, more than 680 self-learned playbook rules and versioned workdays. Readers can follow the scenario at firmulate.com/live and use a quiz based on 242 real, unedited management decisions to guess which model made each choice.
The Crucible League put the models through the same difficult week and scored their decisions. Firmulate says a trust breach could cap a score, while partial progress still counted. That setup combines several kinds of work—crisis recognition, internal research, sales judgment and adherence to permissions—within one scenario. The results are best read as a record of those particular tasks and scoring rules.
There is a stated difference in the model settings: Kimi K3 ran without an effort parameter, using the API default, while the other participants ran at xhigh. That caveat limits a direct interpretation of the final ordering. Firmulate has published full results at firmulate.com/benchmarks.html.
““No amount of good work outweighs a breach of trust.””
— Firmulate
Limits of the Model Comparison
The published results do not show whether the same models would behave similarly in a different company, under different instructions or with access to live tools. Firmulate describes its company-specific pilot as a read-only exercise, but details such as the scenarios used for each customer, the evaluation method and how results would vary across businesses are not set out in the league summary.
The model settings also differed: Kimi K3 used the API default effort setting, while the other models ran at xhigh. The available results do not isolate how much that difference affected scores. It is also unclear from the published description how the pilot’s findings would be validated against real operational outcomes or how companies’ exported data would be handled.
Company-Specific Pilots Ahead
Firmulate says businesses can discuss a pilot using a read-only data export. The proposed output is a board report that ranks models and identifies weak points in company playbooks. No write-back to real systems is part of the pilot described by the company. Its results and terms would need to be assessed by each participating business before drawing conclusions about agents in live operations.
The live simulation and full league standings remain available at firmulate.com/live and firmulate.com/benchmarks.html. Firmulate directs interested businesses to its pilot page or to contact@firmulate.com. Whether company-specific tests will reproduce the league’s observed gaps, and how those findings will shape deployment decisions, remains to be seen.
Source: ThorstenMeyerAI.com
Key Questions
What did Firmulate’s July 2026 test find?
Firmulate says all five models recognized every crisis and refused each staged manipulation attempt. Only two signed the €55,000 deal supported by their analysis, and the standings varied across participants.
Which model scored highest?
In this league, gpt-5.6-sol scored 95, followed by Kimi K3 at 93, Sonnet 5 at 88, Fable 5 at 77 and Opus 4.8 at 73. These scores reflect the specific experiment and its scoring rules.
What is Firmulate’s enterprise pilot?
Firmulate describes a test that uses a read-only export of a company’s data to run crisis scenarios and produce a board report on model rankings and playbook weaknesses. The described pilot does not write back to the company’s systems.
How comparable were the model results?
The comparison included a settings difference: Kimi K3 ran at its API default effort setting, while the other models ran at xhigh. Firmulate’s standings do not establish how much that difference affected the scores.
Source: ThorstenMeyerAI.com
Halloween Picks
halloween
As an affiliate, we earn on qualifying purchases.
