Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The first big AI disaster is yet to happen

seangoedecke.com

Post reply on HN
Print view View original

81–90 of 154 posts

Re: The first big AI disaster is yet to happen

#81

Post by [deleted] »

simonw wrote: post #57
andy99 wrote: post #25

Earlier quoted context omitted.

I'm not sure, I see an argument that they're mostly overblown and a lot would have to be true for one of these proof-of-concepts to translate into a real harm in the wild. I definitely think they need to be taken seriously, just not convinced of the scale of the harm that will be wrought.

Out of interest, when you said "mostly overblown" were you thinking of prompt injection as "people inject bad prompts into an LLM" (I'd call that "jailbreaking") or were you thinking of the thing where an attacker sneaks bad instructions into the model against the wishes of the model's actual user, eg by hiding instructions on a web page that gets summarized or sending them to the victim in an email?

[deleted]

Re: The first big AI disaster is yet to happen

#82

Post by codr7 »

aprilthird2021 wrote: post #65
cypherpunks01 wrote: post #46

AI-directed bombings of persons are already known to have occurred on a large scale, in Gaza. https://www.972mag.com/lavender-ai-israeli-army-gaza/ One source stated that human personnel often served only as a “rubber stamp” for the machine’s decisions, adding that, normally, they would personally devote only about “20 seconds” to each target before authorizing a bombing — just to make sure the Lavender-marked target…

[flagged]

[flagged]

Re: The first big AI disaster is yet to happen

#83

Post by cypherpunks01 »

TrackerFF wrote: post #59

There's a town in northern Norway where the municipality used AI tools / LLMs to aid their process in restructuring schools. So they were going to close and merge a bunch of schools, and pointed to research that supported their plans. But it turns out that the AI they used, had hallucinated this research. The AI had correctly used relevant researchers, but had hallucinated the work itself. It wasn't until a investiga…

You can go a little bigger than a town in northern Norway - the US Department of Health and Human Services is outputting tokens which cite non-existent studies too.

https://www.washingtonpost.com/health/2025/05/29/maha-rfk-jr...

https://archive.is/TFUSl

Re: The first big AI disaster is yet to happen

#84

Post by reasonableklout »

simonw wrote: post #70
reasonableklout wrote: post #67

Earlier quoted context omitted.

It's not the same as "AI replacing humans", but seems clear that the launch of ChatGPT led to dramatic cuts of NLP researchers. [1][2] For instance, at HuggingFace: > A few days after that, Thom Wolf, who was my manager at Hugging Face and also one of the co-founders, messages me, “Hey, can you get on a call with me ASAP?” He told me that they had fired people from the research team and that the rest would either be…

That giant shock to the NLP research community is more than two years old now. It would be interesting to hear from people affected by that today, I'd like to know what they ended up doing in response.

That's true. From the quanta article it looks like many affected researchers went on to work on LLMs successfully.

I do wonder how people are adjusting to similar, ongoing shocks across other industries. For instance, the ArtStation protest against AI art a couple years ago showed widespread hostility towards the technology. In the time since, AI art has only gotten better and been adopted more, but from articles like [1] it doesn't seem like concept artists have much benefitted.

[1]: https://aftermath.site/ai-video-game-development-art-vibe-co...

Re: The first big AI disaster is yet to happen

#85

Post by paxys »

Big AI disasters have most definitely already happened. We are just terrible at identifing them unless there's something exploding (literally) in our face.

As an example, the recent "Make America Healthy Again" report released by the White House and the Health Secretary (RFK) was found to be AI generated, and was full of dubious science and made-up citations. How many people are going to directly die because of this? Definitely more than a plane crash's worth.

Re: The first big AI disaster is yet to happen

#86

Post by BLKNSLVR »

tonymet wrote: post #36

Covid response / lockdown would be the first epistemological catastrophe due to AI . Not strictly due to LLM, but broadly due to poorly configured and likely tampered-with diagnostics signals, wryly inference models that led to global triggering of failsafe's and lockdown response protocols. In the simplest terms, poor diagnostics were wildly extrapolated which triggered excessive response. Phony narratives were ampl…

Strong disagree on AI / LLMs having any involvement whatsoever with Covid response / lockdown.

Off topic:

Strong disagree on the rest of what you said. It's also easy to forget the dangers of Covid pre-vaccine when we're quite deeply post-vaccine now.

Re: The first big AI disaster is yet to happen

#87

Post by thfuran »

andy99 wrote: post #2

While I agree, it won't be a purely "AI" disaster the way a plane crash is a direct consequence of the kinetic and potential energy associated with flying. s/AI/automation/ it's always bad to hook an unchecked algorithm up to something dangerous, it doesn't matter if it's an if statement or a neural net. It will be the hooking up that's responsible more than the AI which will just be what fools someone into thinking…

Yes, the first big AI disaster will probably just be a flavor of gross negligence. But new kinds of tools do create new kinds of negligence.

Re: The first big AI disaster is yet to happen

#88

Post by tonymet »

simonw wrote: post #44
tonymet wrote: post #36

Covid response / lockdown would be the first epistemological catastrophe due to AI . Not strictly due to LLM, but broadly due to poorly configured and likely tampered-with diagnostics signals, wryly inference models that led to global triggering of failsafe's and lockdown response protocols. In the simplest terms, poor diagnostics were wildly extrapolated which triggered excessive response. Phony narratives were ampl…

Covid killed 1,193,165 people in the USA alone. Do you think the lockdowns failed to prevent that number from being significantly larger?

diagnostic error

Re: The first big AI disaster is yet to happen

#89

Post by tonymet »

amanaplanacanal wrote: post #78
tonymet wrote: post #36

Covid response / lockdown would be the first epistemological catastrophe due to AI . Not strictly due to LLM, but broadly due to poorly configured and likely tampered-with diagnostics signals, wryly inference models that led to global triggering of failsafe's and lockdown response protocols. In the simplest terms, poor diagnostics were wildly extrapolated which triggered excessive response. Phony narratives were ampl…

Now I'm wondering if you used an LLM to generate this sci-fi scenario.

more of a model. human generated

Re: The first big AI disaster is yet to happen

#90

Post by thfuran »

teeray wrote: post #71

I think the most plausible AI disaster is going to be of a Therac-25 nature. Some "everyone must use AI" shop will ship an AI-generated feature that some dev signed off on and it might kill people.

People are using LLM slop in legal filings. I'm sure someone somewhere is already using it in worse ways.
Post reply on HN