Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The first big AI disaster is yet to happen

seangoedecke.com

Post reply on HN
Print view View original

41–50 of 154 posts

Re: The first big AI disaster is yet to happen

#41

Post by simonw »

andy99 wrote: post #25
simonw wrote: post #13

Something I find surprising is that we haven't seen a front-page-of-the-news prompt injection attack yet, where vast amounts of important private data were stolen from a company via a classic project injection exfiltration attack. There was a new one of those reported against Microsoft 365 Copilot just today (patched before the vulnerability was shared) - I wrote that up here: https://simonwillison.net/2025/Jun/11/ec…

I'm not sure, I see an argument that they're mostly overblown and a lot would have to be true for one of these proof-of-concepts to translate into a real harm in the wild. I definitely think they need to be taken seriously, just not convinced of the scale of the harm that will be wrought.

An attack where somebody emails the VP of marketing of a company and gets automatically leaked their latest sales figures feels pretty impactful to me.

Or emails the CEO and gets automatically forwarded all of their password reset messages.

Re: The first big AI disaster is yet to happen

#42

Post by [deleted] »

michaelt wrote: post #7

> Chatbot sites like character.ai and Chai AI have been linked to user suicides, for understandable reasons: if you allow users near-unlimited control over chatbots, over time you’ll eventually end up with some chatbots who get into an “encouraging the user to self-harm” state. If cooking hadn't been invented 2.5 million years ago and instead you tried to invent it today, I wonder how many thousands of articles would…

[deleted]

Re: The first big AI disaster is yet to happen

#43

Post by protocolture »

michaelt wrote: post #7

> Chatbot sites like character.ai and Chai AI have been linked to user suicides, for understandable reasons: if you allow users near-unlimited control over chatbots, over time you’ll eventually end up with some chatbots who get into an “encouraging the user to self-harm” state. If cooking hadn't been invented 2.5 million years ago and instead you tried to invent it today, I wonder how many thousands of articles would…

Considering that governments are trying to remove gas stoves from peoples homes for safety reasons yes, you would probably have a hard time introducing them from scratch today.

Re: The first big AI disaster is yet to happen

#44

Post by simonw »

tonymet wrote: post #36

Covid response / lockdown would be the first epistemological catastrophe due to AI . Not strictly due to LLM, but broadly due to poorly configured and likely tampered-with diagnostics signals, wryly inference models that led to global triggering of failsafe's and lockdown response protocols. In the simplest terms, poor diagnostics were wildly extrapolated which triggered excessive response. Phony narratives were ampl…

Covid killed 1,193,165 people in the USA alone. Do you think the lockdowns failed to prevent that number from being significantly larger?

Re: The first big AI disaster is yet to happen

#45

Post by [deleted] »

michaelt wrote: post #7

> Chatbot sites like character.ai and Chai AI have been linked to user suicides, for understandable reasons: if you allow users near-unlimited control over chatbots, over time you’ll eventually end up with some chatbots who get into an “encouraging the user to self-harm” state. If cooking hadn't been invented 2.5 million years ago and instead you tried to invent it today, I wonder how many thousands of articles would…

[deleted]

Re: The first big AI disaster is yet to happen

#46

Post by cypherpunks01 »

AI-directed bombings of persons are already known to have occurred on a large scale, in Gaza.

https://www.972mag.com/lavender-ai-israeli-army-gaza/

One source stated that human personnel often served only as a “rubber stamp” for the machine’s decisions, adding that, normally, they would personally devote only about “20 seconds” to each target before authorizing a bombing — just to make sure the Lavender-marked target is male. This was despite knowing that the system makes what are regarded as “errors” in approximately 10 percent of cases, and is known to occasionally mark individuals who have merely a loose connection to militant groups, or no connection at all.

Re: The first big AI disaster is yet to happen

#47

Post by tptacek »

The downside risks this post talks about are (for obvious reasons) all things that human driven complex systems have already caused, by doing things that were in retrospect stupid. So the basic thesis of this piece is that AI will eventually cause humans to do some kind of stupid stuff, faster and worse. Seems true!

Re: The first big AI disaster is yet to happen

#49

Post by simonw »

icedchai wrote: post #35
LPisGood wrote: post #22

Earlier quoted context omitted.

I don’t think AI is meaningfully taking any jobs. Just like return to office mandates, it is just more cover to do layoffs in less than prosperous economic conditions.

AI may prevent lost jobs from coming back, even if it didn't cause the loss. Companies will be able to do more with less.

> Companies will be able to do more with less.

I guess everyone on Hacker News should stop developing software then, since almost every piece of software I've ever worked on has allowed companies (and individuals) to do more with less.

Re: The first big AI disaster is yet to happen

#50

Post by bsder »

michaelt wrote: post #7

> Chatbot sites like character.ai and Chai AI have been linked to user suicides, for understandable reasons: if you allow users near-unlimited control over chatbots, over time you’ll eventually end up with some chatbots who get into an “encouraging the user to self-harm” state. If cooking hadn't been invented 2.5 million years ago and instead you tried to invent it today, I wonder how many thousands of articles would…

Fire is vitally useful ... Starting a wildfire likely kills your entire tribe.

Now, perhaps there should be some rules around using fire?

Post reply on HN