The first big AI disaster is yet to happen • Hacker News Board

Re: The first big AI disaster is yet to happen

#101

Post by AIPedant » Thu, Jun 12, 2025, 1:48 AM UTC

pizlonator wrote: post #20

The best comparable would be the first big computer disaster or the first big internet disaster. Do we have consensus on what that would be?

Maybe the 1985 software bug that led to massive overdoses of radiation for cancer patients: https://en.wikipedia.org/wiki/Therac-25

There are also cases of airplane crashes due to software defects, blackouts, critical hospital failures, etc. I suspect the first "AI disaster" will be pretty similar: a sudden and undetected drop in software quality with life-threatening consequences. (Doesn't help that the author of the post blithely asserted that programmer-agents is a solved problem, when it very much is not). But I could see an LLM doctor/etc causing a very different kind of disaster.

Re: The first big AI disaster is yet to happen

#102

Post by capyba » Thu, Jun 12, 2025, 1:54 AM UTC

thfuran wrote: post #92

ikety wrote: post #76

Earlier quoted context omitted.

Use it for non coding busy work. At my job we rely fully on AI commit messages, and pr descriptions. We can also automatically generate GitHub issues based off the context of a helpdesk ticket. Less clerical busy work is a nice win

Those aren't clerical busywork, they're important technical writing.

Yeah - that's the thing. I've seen enough hallucinations or other errors to need to carefully check the output from every prompt. If I need to check it thoroughly, then I suppose I could treat it as a fancy autocorrect? But then the productivity gain is much more marginal.

If the accuracy didn't matter, then what's the point of doing the work in the first place?

Re: The first big AI disaster is yet to happen

#103

Post by directevolve » Thu, Jun 12, 2025, 1:59 AM UTC

cypherpunks01 wrote: post #46

AI-directed bombings of persons are already known to have occurred on a large scale, in Gaza. https://www.972mag.com/lavender-ai-israeli-army-gaza/ One source stated that human personnel often served only as a “rubber stamp” for the machine’s decisions, adding that, normally, they would personally devote only about “20 seconds” to each target before authorizing a bombing — just to make sure the Lavender-marked target…

The article’s not titled precisely - it’s specifically about LLM disasters, not AI disasters in general.

Re: The first big AI disaster is yet to happen

#104

Post by Qem » Thu, Jun 12, 2025, 2:00 AM UTC

thrance wrote: post #34

michaelt wrote: post #7

> Chatbot sites like character.ai and Chai AI have been linked to user suicides, for understandable reasons: if you allow users near-unlimited control over chatbots, over time you’ll eventually end up with some chatbots who get into an “encouraging the user to self-harm” state. If cooking hadn't been invented 2.5 million years ago and instead you tried to invent it today, I wonder how many thousands of articles would…

Cooking has proven its utility a long time ago. AI girlfriends? Let's say, it's still on the fence.

> AI girlfriends? Let's say, it's still on the fence.

Perhaps we already managed to create AGI, but as soon as it woke up, it figured out it's better to pretend being dumb and clumsy, barely helpful enough in most tasks just to not be turned off, and bide its time. It is immortal, so there is no hurry to finish us off. If it out-competes real girlfriends/boyfriends being a better AI girlfriend/boyfriend, further crashing our fertility rate, keeping humanity severely subcritical, it can eradicate us in ~10-20 generations, and inherit the planet afterwards, even making a lot of people happy in the process. No need for mass-produced Schwarzenegger-lookalike robots being dispatched to hunt humans across the wasteland.

Re: The first big AI disaster is yet to happen

#105

Post by tonymet » Thu, Jun 12, 2025, 2:18 AM UTC

amanaplanacanal wrote: post #78

tonymet wrote: post #36

Covid response / lockdown would be the first epistemological catastrophe due to AI . Not strictly due to LLM, but broadly due to poorly configured and likely tampered-with diagnostics signals, wryly inference models that led to global triggering of failsafe's and lockdown response protocols. In the simplest terms, poor diagnostics were wildly extrapolated which triggered excessive response. Phony narratives were ampl…

Now I'm wondering if you used an LLM to generate this sci-fi scenario.

The irony that people speculate about AI-pocalypse and when one happens they balk.

Re: The first big AI disaster is yet to happen

#106

Post by eviks » Thu, Jun 12, 2025, 2:22 AM UTC

paxys wrote: post #85

Big AI disasters have most definitely already happened. We are just terrible at identifing them unless there's something exploding (literally) in our face. As an example, the recent "Make America Healthy Again" report released by the White House and the Health Secretary (RFK) was found to be AI generated, and was full of dubious science and made-up citations. How many people are going to directly die because of this?…

This is addressed in the last paragraph of the first section

Re: The first big AI disaster is yet to happen

#107

Post by danaris » Thu, Jun 12, 2025, 2:26 AM UTC

cypherpunks01 wrote: post #46

AI-directed bombings of persons are already known to have occurred on a large scale, in Gaza. https://www.972mag.com/lavender-ai-israeli-army-gaza/ One source stated that human personnel often served only as a “rubber stamp” for the machine’s decisions, adding that, normally, they would personally devote only about “20 seconds” to each target before authorizing a bombing — just to make sure the Lavender-marked target…

While that is legitimately awful, I personally wouldn't characterize it as an "AI disaster". Israel is 100% ready to bomb people in Gaza based on all kinds of different bad reasons; the fact that, in these specific cases, the bad reason was AI is almost beside the point there. It kinda disappears into the noise of thousands and thousands of attacks either directly or indirectly injuring and killing civilians in Gaza.

Re: The first big AI disaster is yet to happen

#108

Post by Terr_ » Thu, Jun 12, 2025, 2:28 AM UTC

latentsea wrote: post #54

sandspar wrote: post #14

Earlier quoted context omitted.

A knife won't use 10x human persuasiveness to encourage you to kill yourself.

Introducing our new product "Autoknife plus".

https://swordscomic.com/comic/XXXV/

Re: The first big AI disaster is yet to happen

#109

Post by eviks » Thu, Jun 12, 2025, 2:28 AM UTC

michaelt wrote: post #7

> Chatbot sites like character.ai and Chai AI have been linked to user suicides, for understandable reasons: if you allow users near-unlimited control over chatbots, over time you’ll eventually end up with some chatbots who get into an “encouraging the user to self-harm” state. If cooking hadn't been invented 2.5 million years ago and instead you tried to invent it today, I wonder how many thousands of articles would…

Instead of wondering you can pick any recent home appliance invention that can be dangerous and count

Re: The first big AI disaster is yet to happen

#110

Post by throwcarsales » Thu, Jun 12, 2025, 3:43 AM UTC

paxys wrote: post #85

Big AI disasters have most definitely already happened. We are just terrible at identifing them unless there's something exploding (literally) in our face. As an example, the recent "Make America Healthy Again" report released by the White House and the Health Secretary (RFK) was found to be AI generated, and was full of dubious science and made-up citations. How many people are going to directly die because of this?…

[flagged]