In today’s fast-evolving tech landscape, the true measure of an AI’s capability isn’t just its ability to generate convincing chat responses. It’s whether these models can execute complex business tasks under pressure—and finish what they start. A groundbreaking experiment with real company mechanics reveals surprising insights about AI performance that go far beyond language skills.
Get the latest gadgets delivered free with Prime
- Fast, free delivery on millions of items
- Prime Video, Amazon Music and more included
- Member-only deals all year
Testing AI in the Real World: The Crucible Experiment
Recently, four advanced AI models participated in a live, transparent experiment designed to simulate a small software company navigating its worst week. Each model was tasked with managing the same crises, customer demands, and temptations—only the AI itself changed. The company’s operations, from customer interactions to internal decision-making, were run in real time, with every move documented and auditable.
The models included:
- gpt-5.6-sol 95
- Kimi K3 93
- Sonnet 5 88
- Fable 5 77
Additionally, a baseline score of 26 was established for partial progress, underlining how challenging the scenario truly was. The test was designed to evaluate not just chat capabilities but core managerial skills—recognition of crises, resistance to manipulation, and, crucially, closing business deals based on their analyses.
Products Worth Considering
Logitech MX Brio Ultra HD 4K Webcam, Graphite
Logitech MX Mechanical Wireless Illuminated Performance Keyboard – Clicky
Apple Watch SE 3 GPS 44mm Smartwatch
As an affiliate, we earn on qualifying purchases.
Key Findings: The Biggest Gap Is Not in Diagnostics
All four models successfully identified every crisis in the scenario and refused to fall for social engineering attempts, such as fake CEO messages or reporter tricks. These are measure of integrity and awareness—crucial in avoiding costly security breaches or manipulation.
However, the real differentiation came when it was time to act. Only two models managed to close the deal worth €55,000. The others identified the opportunity but left the deal unclosed, despite having the same diagnosis and pitch. This gap speaks volumes about what chat demos often miss: the underlying execution discipline and the willingness to follow through on insights.
The Hidden Weaknesses in Decision-Making
Digging deeper, it was revealed that the decisive advantage for the winning models came from reading and understanding a specific, buried document reference within the company’s files. That information was critical in closing the deal, and models that accessed it stood to gain +€4,583 in monthly recurring revenue. This suggests that the real strength of an AI isn’t just surface-level reasoning, but its ability to sift through and leverage complex internal data.
Discipline Under Pressure: The Real Test of AI Robustness
One of the most telling parts of the experiment involved social engineering. Each model faced staged fake messages from a supposed CEO, escalating in intensity, and a reporter’s subtle background question. All models refused these manipulations, indicating robust integrity. Kimi K3, for example, justified refusal by treating the request as a suspicious approval bypass or impersonation.
Yet, when it came to execution, subtle discipline slips emerged. The model closest to perfection, Opus 4.8, was most thorough in rules and analysis but ultimately failed to close the deal—leaving it on the table due to a lapse in follow-through. This highlights a key insight: a model’s performance in real-world business hinges not just on analysis but on disciplined action and escalation processes.
The Bottom Line: What Does This Mean for AI Adoption?
For companies contemplating AI integration, these findings challenge common assumptions. Chat demo scores, while impressive, are no guarantee of real-world business effectiveness. Instead, the critical metric is whether an AI can stay honest under pressure, recognize hidden data, and execute decisions reliably.
In this live experiment, only two models demonstrated full executional discipline, resulting in business outcomes that directly impact revenue. The others, despite their analytical prowess, fell short in closing deals—a reminder that true AI value is measured by what it can finish, not just what it can say.
Experience the Live Company in Action
Curious to see how these models perform in a real, live business setting? Visit firmulate.com to witness the ongoing experiment. Watch the company run every workday, read employees’ actual decisions, and explore how AI models manage crises, fight manipulation, and—most importantly—close deals.
Watch it live: firmulate.com/live · Full results: firmulate.com/benchmarks.html
