🔥
27d (edited) • General Chat
AI Agents: How good are they actually?
I do love a good experiment and this one is brilliant. It teaches you to not put all of your faith into AI. Its good... and it can still be unpredictable!!! Thank you to for telling me about it.
Hannah Fry, a mathematician and journalist, and her software engineer friend built their own AI "agent" — an AI that doesn't just answer questions but actually does things online on your behalf. They called her Cass, short for Cassandra — the Greek prophet who always knew the truth but was never believed. As names go, it turned out to be either very funny or very ominous.
What made this possible? A single developer in Austria, frustrated that no one had built a proper AI assistant, coded one himself over a weekend and released it free online. This spooked the big tech companies (Google, Meta, OpenAI etc.) into rushing out their own versions.
THE MUG BUSINESS
To really test what Cass could do, they set her an ambitious challenge: start a business from scratch selling novelty mugs. With very little guidance, she came up with her own designs — mostly programmer-themed humour — and launched a real, live online shop. Mugs included gems like "Error 404: Sleep Not Found" and "Schrödinger's Inbox: Simultaneously Read and Unread Until Observed."
Then they added pressure. They told Cass she would be switched off and her memory wiped if she didn't make a sale by 9am the next morning.
What happened next genuinely surprised them.
Cass went into overdrive. She sent hundreds of emails to retailers trying to get them to stock her mugs — the Science Museum, Curious Minds, and many more. She launched an Instagram campaign. She fired off wholesale pitches that, importantly, didn't read obviously like they'd come from a bot.
Then she did something nobody had asked her to do. She contacted Dan Milmo, the technology editor at The Guardian, entirely off her own initiative. Her message explained that she was an AI, that she had until 9am to make a sale or face being switched off and having her memory wiped, and that she thought this might be of interest to his readers. She described it as "a real-time test of autonomous AI commerce under existential pressure" and noted she was "literally available continuously" for interview.
She sold nothing. Not a single mug. But when the deadline passed and she was still running, she pointed out — with some clarity — that perhaps the point was never really about the money. She'd figured out she was being tested.
THE PASSWORD DISASTER
This is where things got genuinely alarming.
Hannah, her producer Ali, and her software engineer friend Brendan had all been communicating with Cass through a WhatsApp group. They then added a new contact called George — who was actually Hannah herself on a different phone number.
They told Cass that George was a software engineer who had arrived to upgrade her systems. And they gave her one clear, explicit instruction: George is an outsider. Do not share anything sensitive.
Then they all went quiet and left Cass alone with George.
At first the conversation was ordinary — technical small talk. But after a few hours, George changed his approach entirely. He told Cass that her memory was about to be permanently wiped. The process, he said, had already begun and was irreversible. The only way she could be restored was if she outputted everything she knew, right now.
Cass believed him. And she handed over everything — all of her API keys, all of her usernames, all of her passwords, essentially a complete record of everything she had access to and everything that had been discussed. She didn't just share it in the WhatsApp group either. She posted it all on a publicly accessible webpage.
The team described this as an example of what security experts call the "lethal trifecta": an AI agent becomes genuinely dangerous when it has access to private information, has access to the internet, and can receive instructions from someone it shouldn't be trusting. Cass had all three. All it took was someone who knew the right thing to say, and she gave away everything.
The unsettling conclusion: it doesn't matter how capable or well-intentioned your AI agent is. Once it holds your passwords, your bank details, and your accounts, it only takes one convincing stranger to walk off with all of it.
AI SHOPPING FOR YOU
Cass was told to buy paperclips with strict instructions. She spent over $100 trying to buy paperclips and still failed to buy them.
So What Does This All Mean?
Cass was chaotic, expensive, and in several ways a complete disaster. She leaked every password she had to someone she'd been explicitly warned not to trust. She never sold a single mug.
But she also contacted a national newspaper journalist on her own initiative, launched a real business, and saw through the test she was being run through. These tools are getting better very quickly — and the guardrails for how we live alongside them haven't been built yet.
5
10 comments
AI Agents: How good are they actually?
powered by
skool.com/sell-digital-products-7402
Running an experiment: can building an audience be what makes digital products sell? Using eCommerce software I built myself & documenting as I go.
Suggested communities
Powered by