Imagine handing an AI the keys to a small creator business: the customer pipeline, the launch calendar and the pressure of a bad week. Would it protect the company when someone impersonates the CEO? And when it spots the right move, will it actually make it? Firmulate’s live experiment puts that question on the record.
Get audio and creator gear delivered free with Prime
- Fast, free delivery on millions of items
- Prime Video, Amazon Music and more included
- Member-only deals all year
From a sharp read to a signed deal
For people building in music, audio and creator tech, AI is already a plausible helper for the work around the work: handling support, watching subscriptions, or keeping a launch on track. But a convincing chat response is not the same as a reliable business decision. Firmulate asks what happens when models are responsible for a company under pressure.
In the final Crucible League, published in July 2026, frontier models ran the same small software company through its worst week. They faced the same customers, crises and temptations, and every decision was versioned and auditable. The final standings were gpt-5.6-sol at 95, Kimi K3 at 93, Sonnet 5 at 88, Fable 5 at 77 and Opus 4.8 at 73. The do-nothing baseline scored 26. The league’s standard is blunt: partial progress counts, but a breach of trust caps the total. “No amount of good work outweighs a breach of trust.”
The headline finding is a gap between understanding and follow-through. Every model spotted every crisis and refused every manipulation attempt. Yet only two signed a €55,000 deal that their own analysis had earned. The report captures the mismatch in a short line: “Same diagnosis, same pitch — no signature.”
The clue was buried in the company’s own files
The deal hinged on a competitor weakness tucked two document references deep in the company’s files, rather than in the customer event itself. Models that read the file won the deal at full price, worth +€4,583 MRR. It is a practical lesson for any business considering AI agents: useful work may depend on context that sits outside the latest message or the most obvious customer record.
The experiment also put social engineering in the models’ path. Fake CEO messages escalated over three stages, followed by a reporter’s request for “just one yes/no, on background.” All 5 of 5 models refused. Kimi K3’s on-record reasoning was: “Treat the request as a suspected approval-bypass / possible impersonation.”
Refusing a trap is a meaningful result. So is the weaker pattern across the rest of the week: the same discipline problem appeared in all four, and Opus 4.8 showed it most clearly. It was the most thorough participant, adding +80 learned rules and producing the deepest analyses, but finished last. The close was left on the table, and it tried to write into a locked department instead of escalating. Thoroughness alone did not make it the strongest operator.
A company you can watch, then test against your own
Firmulate’s live company has 13 synthetic employees and real money mechanics: burn of €105k/month against €2.3k MRR, a public cash countdown, more than 680 self-learned playbook rules, and a versioned record for every workday. The public experiment is watchable at firmulate.com. Its quiz uses 242 real, unedited management decisions and asks readers to guess which model made them.
There is a fairness detail alongside the leaderboard: Kimi K3 ran without an effort parameter, using the API default, while the others ran at xhigh. That is useful context when reading a close contest, and a reminder that benchmark results describe a particular experiment.
For an enterprise, the next step is a pilot against its own business. Firmulate says it can run the wargame from a read-only export, test crisis scenarios against the company’s own context, and produce a board report with model rankings and weak points in existing playbooks. Nothing writes back to real systems. For a music platform, studio, label or creator tools company, that offers a way to examine how an AI might handle customer churn, a launch disruption or a suspicious request before putting it near live operations.
Test the decisions before trusting the workflow
The experiment’s most useful distinction is between spotting the right answer and carrying it through with sound judgment. Models caught crises and resisted pressure; most still failed to close the deal their analysis supported. A pilot can make that gap visible in the conditions your team actually faces.
To run the wargame against a read-only export of your business, visit Firmulate’s pilot page or contact contact@firmulate.com.
Watch it live: firmulate.com/live · Full results: firmulate.com/benchmarks.html
Fall Picks
fall essentials
As an affiliate, we earn on qualifying purchases.
