Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The first big AI disaster is yet to happen

seangoedecke.com

Post reply on HN
Print view View original

21–30 of 154 posts

Re: The first big AI disaster is yet to happen

#21

Post by bluefirebrand »

michaelt wrote: post #7

> Chatbot sites like character.ai and Chai AI have been linked to user suicides, for understandable reasons: if you allow users near-unlimited control over chatbots, over time you’ll eventually end up with some chatbots who get into an “encouraging the user to self-harm” state. If cooking hadn't been invented 2.5 million years ago and instead you tried to invent it today, I wonder how many thousands of articles would…

Every advancement comes with tradeoffs

We, as a society, should have some amount of caution about embracing advancements that benefit an extremely small portion of people at the expense of the vast majority

I hate to say it but AI absolutely falls into that category

It will benefit corporations, wealthy, executives, and it will absolutely cost the lower and middle class

This is the trajectory we've been on for a decades now. AI seems poised to accelerate that problem even more

So genuinely: is the economic benefit of AI actually going to be net good after we see how the likely social unrest plays out?

Re: The first big AI disaster is yet to happen

#22

Post by LPisGood »

pcurve wrote: post #17

I feel like the first big disaster already happened. In the job market. Anyway, where there is public safety risk involved, I can't imagine a scenario where AI will directly cause a catastrophe. If anything, I feel overall safety will increase. However, I can't help but wonder what the over reliance would do to general population over the next few decades. Would people essentially become dumber and less skilled?

I don’t think AI is meaningfully taking any jobs. Just like return to office mandates, it is just more cover to do layoffs in less than prosperous economic conditions.

Re: The first big AI disaster is yet to happen

#24

Post by LPisGood »

sandspar wrote: post #19
simonw wrote: post #13

Something I find surprising is that we haven't seen a front-page-of-the-news prompt injection attack yet, where vast amounts of important private data were stolen from a company via a classic project injection exfiltration attack. There was a new one of those reported against Microsoft 365 Copilot just today (patched before the vulnerability was shared) - I wrote that up here: https://simonwillison.net/2025/Jun/11/ec…

Sooner or later there'll be a searchable database where you search by name and get their porn history or some other embarassing information.

I mean this was a threat before - google, ISPs, and adult sites that require payment have been places one could attack to make this for decades.

Re: The first big AI disaster is yet to happen

#25

Post by andy99 »

simonw wrote: post #13

Something I find surprising is that we haven't seen a front-page-of-the-news prompt injection attack yet, where vast amounts of important private data were stolen from a company via a classic project injection exfiltration attack. There was a new one of those reported against Microsoft 365 Copilot just today (patched before the vulnerability was shared) - I wrote that up here: https://simonwillison.net/2025/Jun/11/ec…

I'm not sure, I see an argument that they're mostly overblown and a lot would have to be true for one of these proof-of-concepts to translate into a real harm in the wild.

I definitely think they need to be taken seriously, just not convinced of the scale of the harm that will be wrought.

Re: The first big AI disaster is yet to happen

#26

Post by bigfatkitten »

LPisGood wrote: post #22
pcurve wrote: post #17

I feel like the first big disaster already happened. In the job market. Anyway, where there is public safety risk involved, I can't imagine a scenario where AI will directly cause a catastrophe. If anything, I feel overall safety will increase. However, I can't help but wonder what the over reliance would do to general population over the next few decades. Would people essentially become dumber and less skilled?

I don’t think AI is meaningfully taking any jobs. Just like return to office mandates, it is just more cover to do layoffs in less than prosperous economic conditions.

[dead]

Re: The first big AI disaster is yet to happen

#27

Post by jongjong »

I think the first big AI disaster almost certainly did happen already but we don't know about it and will likely never know because of media filter bubbles which the AI creates. Disaster news about AI will never propagate because this news could pose a threat to AI. I think any AI disaster will yield hundreds of alternative explanations. It's trivial to hide reality beneath layer upon layer of complexity and narratives.

The global monetary system has an uncanny ability to silently destroy people who sit on the periphery of politics. This is the perfect mechanism to destroy humanity... Keep silently taking out the people on the periphery, keep making the people in the center more greedy and narcissistic so they don't notice everyone falling all around them and keep supporting the current system... Eventually, those who were near the center will fall victim too, but by that point, they will have no power and no voice and the people left in the center will be yet more greedy and narcissistic and will basically not care.

Re: The first big AI disaster is yet to happen

#28

Post by soulofmischief »

ahmeni wrote: post #11

The first big AI disaster has already happened and it is the thousand little cuts from people putting faith into systems that aren't built to be trustworthy. It is the software dev missing a critical safety check because "ai coding tools" at work have promised 10x delivery results and now they must use them or be seen as falling behind. It is the engineering manager throwing concrete flexural strength questions into…

This is the research part of research and development, this time though it's sort of like a distributed Manhattan project happening in the open. Research takes a lot of money. Anyone can jump in and help improve these models and the ecosystem.

Re: The first big AI disaster is yet to happen

#30

Post by [deleted] »

ahmeni wrote: post #11

The first big AI disaster has already happened and it is the thousand little cuts from people putting faith into systems that aren't built to be trustworthy. It is the software dev missing a critical safety check because "ai coding tools" at work have promised 10x delivery results and now they must use them or be seen as falling behind. It is the engineering manager throwing concrete flexural strength questions into…

[deleted]
Post reply on HN