Imagine managing your pool service business while facing a string of unforeseen crises—customer complaints, supplier issues, and financial pressures—all at once. Now, picture having a digital assistant that not only alerts you to these problems but also makes critical decisions under pressure. The question is: can AI truly handle the complexities of real-world management, especially when the stakes are high? Recent experiments show that while AI can identify crises and resist manipulation, its ability to close deals and follow through remains a hidden strength—or weakness.
Get pool and patio gear delivered free with Prime
- Fast, free delivery on millions of items
- Prime Video, Amazon Music and more included
- Member-only deals all year
Testing AI in the Wild: A Business Simulation
In a groundbreaking live experiment, four advanced AI models were tasked with running a small software company through its worst week. This wasn’t just a test of chat prowess; it was a full-scale simulation where each AI faced the same challenges—customer crises, internal decisions, and manipulative social engineering attempts. The goal? Measure their ability to diagnose problems, resist deception, and ultimately, close a critical €55,000 deal earned through their own analysis.
The Benchmarks and Results
- Among the models, GPT-5.6-SOL scored the highest at 95, followed closely by Kimi K3 at 93, Sonnet 5 at 88, and Fable 5 at 77. A baseline score of 26 represented a do-nothing approach, emphasizing how much progress these models demonstrated.
- All four AI systems identified every crisis and refused manipulation attempts, including staged social engineering scenarios involving fake CEO messages and reporter tricks. For example, Kimi K3 explained its refusal, citing concerns about impersonation and approval bypass.
- Despite this, only two models—GPT-5.6-SOL and Kimi K3—actually signed the deal, earning the €55,000. The others, like Opus 4.8 and Fable 5, detected the issues but failed to follow through, leaving the deal unexecuted.
The Hidden Weakness: Reading the Files
The real difference was in the models’ ability to leverage internal company documents. The decisive advantage went to those that read deeper into the company’s own files—two document references deep—to find critical information that sealed the deal. This underscores a vital point: surface-level chat demos may not reveal an AI’s true operational capabilities. It’s what the AI does when digging into context that counts.
Resisting Social Engineering and Maintaining Discipline
The experiment also tested whether the models could resist social engineering—the fake CEO messages escalating in stages and a reporter request asking for a one-word approval. All models refused these manipulative attempts, demonstrating strong integrity under pressure. Kimi K3 justified its refusal with a clear reasoning: treat the request as a suspected impersonation.
The Live Company and Real Stakes
The company in question isn’t just a simulation; it’s a real business with 13 synthetic employees, daily money mechanics, and a public cash countdown. It burns €105,000 each month against a monthly recurring revenue of €2,300, making the ability to manage crises and close deals under pressure critical. The entire operation is versioned daily, and stakeholders can watch the experiment unfold at firmulate.com/live.
Products Worth Considering
Apple AirPods Pro 3 Wireless Earbuds with Active Noise Cancellation
Amazon Kindle Paperwhite Signature Edition (12th Generation, 2024), 32 GB, Metallic Black
Amazon Kindle Paperwhite (2024), 16GB, 7-inch Display, Black
As an affiliate, we earn on qualifying purchases.
Lessons for Business and AI Investment
This experiment reveals an essential insight: the true measure of an AI’s management capability isn’t just how well it responds to questions or chat prompts. It’s whether it can finish what it starts—reading relevant documents, making decisions, and closing deals—under conditions that mimic real business pressures. The models’ scores and behaviors indicate that while all can detect problems and resist deception, only some can translate diagnosis into action.
What This Means for Pool and Water Lifestyle Businesses
If your business relies on AI to support customer service, sales, or operations, it’s crucial to evaluate whether your AI tool can go beyond surface interactions. Can it read your internal documentation? Will it follow through on commitments? The experiment underscores that the ability to execute, not just to analyze or chat, is what separates merely good AI demos from truly reliable management partners.
Watch it live: firmulate.com/live · Full results: firmulate.com/benchmarks.html
