The first big AI disaster has already happened and it is the thousand little cuts from people putting faith into systems that aren't built to be trustworthy. It is the software dev missing a critical safety check because "ai coding tools" at work have promised 10x delivery results and now they must use them or be seen as falling behind. It is the engineering manager throwing concrete flexural strength questions into the unaccountable void of ChatGPT because he doesn't want to pay for the overtime of asking someone accountable. It is the foolhardy amount of money being shoveled into the entire ecosystem that absolutely cannot provide the returns people need for any of it to make financial sense.
The first big AI disaster is yet to happen
11–20 of 154 posts
Re: The first big AI disaster is yet to happen
#12Dunno what it'll be but I'm positive that we'll be surprised. Grim news like this is always a surprise. The most common headline is "Random city has weird disaster", stuff that you'd never guess ahead of time.
Re: The first big AI disaster is yet to happen
#13Something I find surprising is that we haven't seen a front-page-of-the-news prompt injection attack yet, where vast amounts of important private data were stolen from a company via a classic project injection exfiltration attack.
There was a new one of those reported against Microsoft 365 Copilot just today (patched before the vulnerability was shared) - I wrote that up here: https://simonwillison.net/2025/Jun/11/echoleak/
My suspicion is that the threat of these exfiltration attacks won't be taken seriously until someone gets very publicly burned by one.
Re: The first big AI disaster is yet to happen
#14> Chatbot sites like character.ai and Chai AI have been linked to user suicides, for understandable reasons: if you allow users near-unlimited control over chatbots, over time you’ll eventually end up with some chatbots who get into an “encouraging the user to self-harm” state. If cooking hadn't been invented 2.5 million years ago and instead you tried to invent it today, I wonder how many thousands of articles would…
A knife won't use 10x human persuasiveness to encourage you to kill yourself.
Re: The first big AI disaster is yet to happen
#15The problem with AI is that we might get an artificial general superintelligence before we get the first big disaster, and when a big disaster happens at that point, it could be so big that it’s catastrophic for humanity. Lots has been written about this topic, https://www.lesswrong.com/posts/j9Q8bRmwCgXRYAgcJ/miri-annou... is one of the more pessimistic takes. I wrote a more neutral introduction a while back at http…
Interesting how this has been downvoted to grey. Is it just me or have AI doomers become unfashionable over the last 3-6 months or so, as in rather quickly? OP I'm curious if you've noticed people's responses to you change recently.
Re: The first big AI disaster is yet to happen
#16Reminds me of this supposed 1979 IBM warning https://x.com/bumblebike/status/832394003492564993
It's good advice, but IBM also had dishonorable reasons to assert that the computer (IBM) can't be held accountable.
https://en.wikipedia.org/wiki/IBM_and_World_War_II#Criticism...
Re: The first big AI disaster is yet to happen
#17I feel like the first big disaster already happened. In the job market.
Anyway, where there is public safety risk involved, I can't imagine a scenario where AI will directly cause a catastrophe. If anything, I feel overall safety will increase.
However, I can't help but wonder what the over reliance would do to general population over the next few decades. Would people essentially become dumber and less skilled?
Re: The first big AI disaster is yet to happen
#18“grok, how do i land this plane?”
Re: The first big AI disaster is yet to happen
#19Something I find surprising is that we haven't seen a front-page-of-the-news prompt injection attack yet, where vast amounts of important private data were stolen from a company via a classic project injection exfiltration attack. There was a new one of those reported against Microsoft 365 Copilot just today (patched before the vulnerability was shared) - I wrote that up here: https://simonwillison.net/2025/Jun/11/ec…
Sooner or later there'll be a searchable database where you search by name and get their porn history or some other embarassing information.
Re: The first big AI disaster is yet to happen
#20The best comparable would be the first big computer disaster or the first big internet disaster.
Do we have consensus on what that would be?