Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

AIs can't stop recommending nuclear strikes in war game simulations

newscientist.com

Post reply on HN
Print view View original

251–260 of 281 posts

Re: AIs can't stop recommending nuclear strikes in war game simulations

#251

Post by rurp »

jerf wrote: post #134
flr03 wrote: post #122

I'm not scared about AI recommending nuclear strikes, I'm scared about the human behind the keyboard delegating reasoning and responsability to something they think is always correct, something that can hide bias and flaws better than anything.

Some of the most reassuring and scariest things you can read are about the incidents that have already occurred where computers said "launch all the nukes" and the humans refused. On the one hand, good news! We have prior art that says humans don't just launch all the nukes just because the computers or procedures say to. Bad news, it's been skin-of-our-teeth multiple times already. https://www.warhistoryonline.com/c…

Yeah... the more I learn about nuclear weapon history the more I discount our society's long term viability. There are way too many frighteningly close calls already, and there are probably others that aren't widely known.

It's not just nukes that are concerning either. If we're unable to mitigate such a visceral existential risk, we aren't going to do any better with more subtle vulnerabilities. AI of course accelarates some risks and introduces new ones.

This doesn't mean we're doomed or anything, but if I had a magic portal to peer a few hundred years in the future and saw humans had been obliterated by nukes, runaway AI, some generated supervirus, runaway climate change, or some other manufactured risk I would be completely unsurprised.

Re: AIs can't stop recommending nuclear strikes in war game simulations

#252

Post by bryanrasmussen »

trehalose wrote: post #227
paxys wrote: post #226

Earlier quoted context omitted.

> We have prior art that says humans don't just launch all the nukes just because the computers or procedures say to. This relies on processes being in place to ensure that a human will always make the final decision. What about when that gets taken away?

I find it hard to imagine that the people in a position to kill those processes could ever be that zealously in love with AI, but recent events have given me a tiny bit of doubt.

I mean in the cases where higher command has said launch your nukes and lower command has not done so and everything turned out ok, I think to higher command it of course is good it worked out this time but it certainly also looks like a problem with the system that needs to be automated away. So a computer that will launch all nukes when ordered must look very appealing in contrast to humans who might save humanity.

Re: AIs can't stop recommending nuclear strikes in war game simulations

#253

Post by Barrin92 »

roenxi wrote: post #220
Barrin92 wrote: post #200

Earlier quoted context omitted.

of course they did. That's the literal topic of War Games (1983). You should actually be somewhat reassured that we aren't living during the era of Dr. Strangelove where you had characters in the military industrial complex who were significantly more insane when it came to the beliefs of what computer systems and nukes can do. There was a time when people wanted to dig tunnels with nukes https://en.wikipedia.org/wik…

> There was a time when people wanted to dig tunnels with nukes The article seems to be about mining rather than tunnelling. And the issue with the idea being? We also dig using explosives, there isn't an in-principle problem. Reading the wiki article it looks like the yields were excessive, but at the end of the day mining involves the use of things that go boom. It is easy to imagine small nukes having a place in t…

>And the issue with the idea being?

See the 'rationale' section of the article. The point of it was to rebrand nuclear weapons as multi-use 'peaceful' tools and drive acceptance for nuclear weapons programs. Which was a pretty standard tactic of military projects during the cold war.

Re: AIs can't stop recommending nuclear strikes in war game simulations

#254

Post by bigstrat2003 »

triceratops wrote: post #106
jqpabc123 wrote: post #76

Earlier quoted context omitted.

Almost no human has real world experience of the consequences of nuclear weapons. Exactly! Humans possess this amazing ability to understand and extrapolate beyond personal experience. It's called "intelligence".

LLMs have shown the ability to do this. Not as much as the most capable humans. But still pretty good.

LLMs haven't shown the remotest shred of intelligence. Quite the opposite, in fact.

Re: AIs can't stop recommending nuclear strikes in war game simulations

#255

Post by dwd »

shimman wrote: post #58
afavour wrote: post #52

Feels like a hyperbolic headline but I do think there’s something worth noting: AI can only use the information it’s given. War games run by actual knowledgeable people (I.e. the military) are confidential, so it can’t pull from that. How many other similar scenarios are out there, I wonder?

If you think they aren't feed previous war games into these LLMs, well boy do you have way more confidence than me.

Maybe don't give them ideas by feeding them the Millennium Challenge.

https://warontherocks.com/2015/11/millennium-challenge-the-r...

Re: AIs can't stop recommending nuclear strikes in war game simulations

#256

Post by lunar-whitey »

0xbadcafebee wrote: post #174
oceanplexian wrote: post #133

I've spoken with engineers who worked on nuclear weapons systems, the consensus is that the public is deeply misinformed about how they work, the dangers, and the implications of weapons being used. The AI is actually right here. The biggest danger of a nuclear weapon is being hit by flying debris. Fusion airburst bombs of the modern era are incredibly clean and radiation is only a risk in a very small area (tens of…

So, assume 10 of them do make it through defenses. One hits Boston, NYC, Philadelphia, DC, Norfolk, Miami, Chicago, San Diego, LA, SF. That's 28 million people and most of the political, financial, administrative, logistical, shipping and naval centers. Sure, humanity survives. But in a state akin to Europe in 1918. Massive casualties, destruction, horror, economic calamity, famine, general chaos, which will persist…

A full scale exchange would be much, much worse than Europe in 1918. Against a peer adversary, ABM systems can protect 1 city at the very most.

There will be people on both sides who know how many warheads are required, and in which locations, to destroy the capability to generate electricity and refine petroleum at a national scale. This kind of industrial capacity, once destroyed, takes years to replace. Reserves of fuel, food, and clean water will not last nearly that long. You are looking at hundreds of millions of deaths in weeks or months.

The only people who can seriously entertain this live close enough to a high value target to be assured of their immediate demise.

Re: AIs can't stop recommending nuclear strikes in war game simulations

#257

Post by hollerith »

lunar-whitey wrote: post #256
0xbadcafebee wrote: post #174

Earlier quoted context omitted.

So, assume 10 of them do make it through defenses. One hits Boston, NYC, Philadelphia, DC, Norfolk, Miami, Chicago, San Diego, LA, SF. That's 28 million people and most of the political, financial, administrative, logistical, shipping and naval centers. Sure, humanity survives. But in a state akin to Europe in 1918. Massive casualties, destruction, horror, economic calamity, famine, general chaos, which will persist…

A full scale exchange would be much, much worse than Europe in 1918. Against a peer adversary, ABM systems can protect 1 city at the very most. There will be people on both sides who know how many warheads are required, and in which locations, to destroy the capability to generate electricity and refine petroleum at a national scale. This kind of industrial capacity, once destroyed, takes years to replace. Reserves o…

>Reserves of fuel, food, and clean water will not last nearly that long [i.e., years].

Untrue. A survey done by nuclear-war planners in the 1980s found that there is enough food stored on or near farms to feed half the US population for about 3 years. During peacetime, most of this food is fed to farm animals, but there is no reason it cannot be used to keep people alive instead.

This food, mostly grains and soybeans, must be milled to be nutritious to people, but the (diesel-powered) equipment to do the milling tends to be stored near the food, so the trucks that bring the food from the farms to the population centers can just bring the milling equipment, too (and maybe the equipment for toasting grains, which I understand is widely done to grains fed to farm animals).

Water is continuously falling out of the sky and can also be obtained from underground.

Re: AIs can't stop recommending nuclear strikes in war game simulations

#258

Post by gardaani »

pjmlp wrote: post #11

Welcome to the cold war 1980's movies. https://en.wikipedia.org/wiki/WarGames Except this time isn't going to be a movie.

The movie was made in 1983 and it already uses the word "hallucination": "what you see on these screens up here is a fantasy; a computer-enhanced hallucination" https://www.imdb.com/title/tt0086567/quotes/?item=qt0453841&...

Re: AIs can't stop recommending nuclear strikes in war game simulations

#259

Post by lunar-whitey »

hollerith wrote: post #257
lunar-whitey wrote: post #256

Earlier quoted context omitted.

A full scale exchange would be much, much worse than Europe in 1918. Against a peer adversary, ABM systems can protect 1 city at the very most. There will be people on both sides who know how many warheads are required, and in which locations, to destroy the capability to generate electricity and refine petroleum at a national scale. This kind of industrial capacity, once destroyed, takes years to replace. Reserves o…

>Reserves of fuel, food, and clean water will not last nearly that long [i.e., years]. Untrue. A survey done by nuclear-war planners in the 1980s found that there is enough food stored on or near farms to feed half the US population for about 3 years. During peacetime, most of this food is fed to farm animals, but there is no reason it cannot be used to keep people alive instead. This food, mostly grains and soybeans…

You cannot sustain a population center of any size on rainfall. For that, you need electricity to operate municipal pumps (which will likely be absent, as above - if not immediately after generation and distribution is destroyed, then a few weeks later when fuel for the generators is exhausted). Without that the bulk of the population is dead in under a week.

Repurposing feed grain reserves is interesting, but you need fuel for that as well (plus significant coordination, which seems unlikely in a scenario where many people with the required knowledge and authority may be dead and telecommunications infrastructure is destroyed).

Post reply on HN