Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

AIs can't stop recommending nuclear strikes in war game simulations

newscientist.com

Post reply on HN
Print view View original

131–140 of 281 posts

Re: AIs can't stop recommending nuclear strikes in war game simulations

#131

Post by [deleted] »

techblueberry wrote: post #14
jqpabc123 wrote: post #5

Why is this surprising? Nuclear weapons are available. AI has limited real world experience or grasp of the consequences. Nuke 'em seems like the obvious choice --- for something with a grade school mentality. Similar deficits in reasoning are manifested in AI results every day. Let's fire 'em and hire AI seems like the obvious choice --- for someone with a grade school mentality and blinded by greed.

There was a recent conflict that came up, and there was a debate about whether or not one of the sides was committing war crimes. And I remember thinking to myself and saying in the debate “if this were a video game strategically speaking, I’d be committing war crimes.” And sadly, I think this logic holds up.

[deleted]

Re: AIs can't stop recommending nuclear strikes in war game simulations

#132

Post by pibaker »

jqpabc123 wrote: post #5

Why is this surprising? Nuclear weapons are available. AI has limited real world experience or grasp of the consequences. Nuke 'em seems like the obvious choice --- for something with a grade school mentality. Similar deficits in reasoning are manifested in AI results every day. Let's fire 'em and hire AI seems like the obvious choice --- for someone with a grade school mentality and blinded by greed.

I feel this reflects a deeper problem with letting AI do any kind of decision making. They have no real world experience. They feel no real world consequences. They have no real stake in any decision they make.

Human societies get to control their members' actions by imposing real life consequences. A company can fire you, a partner can divorce you, the state can jail you, the public can shame you. None of these works on the current crop of LLM based AI systems, which as far as I can tell are only trained to handle very narrow tasks where they don't need to even worry about keeping themselves alive. How do you make AIs work in a society? I don't know. Maybe the best move is to not play the game.

Re: AIs can't stop recommending nuclear strikes in war game simulations

#133

Post by oceanplexian »

I've spoken with engineers who worked on nuclear weapons systems, the consensus is that the public is deeply misinformed about how they work, the dangers, and the implications of weapons being used. The AI is actually right here.

The biggest danger of a nuclear weapon is being hit by flying debris.

Fusion airburst bombs of the modern era are incredibly clean and radiation is only a risk in a very small area (tens of miles) for a short time (days to weeks). In a modern conflict a significant fraction of nukes would be intercepted before they reached the United States. There are far fewer of them than there were in the 1980s (A few 1000's vs 40,000). Most would be used on strategic military targets, ships, bases, etc. Not to say it would be a good time, but it wouldn't be the "end of humanity" or anything even remotely like it.

Re: AIs can't stop recommending nuclear strikes in war game simulations

#134

Post by jerf »

flr03 wrote: post #122

I'm not scared about AI recommending nuclear strikes, I'm scared about the human behind the keyboard delegating reasoning and responsability to something they think is always correct, something that can hide bias and flaws better than anything.

Some of the most reassuring and scariest things you can read are about the incidents that have already occurred where computers said "launch all the nukes" and the humans refused. On the one hand, good news! We have prior art that says humans don't just launch all the nukes just because the computers or procedures say to. Bad news, it's been skin-of-our-teeth multiple times already.

https://www.warhistoryonline.com/cold-war/refused-to-launch-... - This isn't even the incident I was searching for to reference! This one was news to me.

https://en.wikipedia.org/wiki/Stanislav_Petrov#Incident - This is the one I was looking for.

Re: AIs can't stop recommending nuclear strikes in war game simulations

#135

Post by triceratops »

jqpabc123 wrote: post #119
triceratops wrote: post #109

Earlier quoted context omitted.

No. That's why I'm asking where it comes from. The explanation that "LLMs don't have experience of nuclear war" isn't satisfying because nobody really has any experience of nuclear war.

Humans don't really need to experience nuclear war to comprehend the consequences and implications of it. LLMs don't really comprehend much of anything. It just looks at what is in it's training database and tries to find similar questions or discussion in order to assemble a plausible sounding answer based on probability. Not the sort of thing anyone should rely on for "critical" decision making.

> It just looks at what is in it's training database and tries to find similar questions or discussion

I feel like we're going around in circles here. So I'll try to explain one last time.

Most of the content about nuclear war in any LLM's training set is almost surely about how horrifying it is and how we must never engage in it. Because that's what humans usually say about nuclear war. The plausible sounding answer about nuclear war, based on probability, really should be "don't do it". So why isn't it?

Re: AIs can't stop recommending nuclear strikes in war game simulations

#136

Post by cantalopes »

oceanplexian wrote: post #133

I've spoken with engineers who worked on nuclear weapons systems, the consensus is that the public is deeply misinformed about how they work, the dangers, and the implications of weapons being used. The AI is actually right here. The biggest danger of a nuclear weapon is being hit by flying debris. Fusion airburst bombs of the modern era are incredibly clean and radiation is only a risk in a very small area (tens of…

[flagged]

Re: AIs can't stop recommending nuclear strikes in war game simulations

#137

Post by jhallenworld »

mrlonglong wrote: post #120

WOPR was the first fictional AI to realise to win is not to play at all. From the War Games (1983) film.

Colossus/Guardian was the first AI to realize that the humans could be easily coerced by using their own nukes against them.

From the Colossus: The Forbin Project (1970) film.

Re: AIs can't stop recommending nuclear strikes in war game simulations

#138

Post by slopinthebag »

cantalopes wrote: post #136
oceanplexian wrote: post #133

I've spoken with engineers who worked on nuclear weapons systems, the consensus is that the public is deeply misinformed about how they work, the dangers, and the implications of weapons being used. The AI is actually right here. The biggest danger of a nuclear weapon is being hit by flying debris. Fusion airburst bombs of the modern era are incredibly clean and radiation is only a risk in a very small area (tens of…

[flagged]

Nah they actually sound reassuring, I don't trust them but I would like to believe if some crazy president decided to start a nuclear war it wouldn't be the end of humanity.

Re: AIs can't stop recommending nuclear strikes in war game simulations

#139

Post by saidnooneever »

oceanplexian wrote: post #133

I've spoken with engineers who worked on nuclear weapons systems, the consensus is that the public is deeply misinformed about how they work, the dangers, and the implications of weapons being used. The AI is actually right here. The biggest danger of a nuclear weapon is being hit by flying debris. Fusion airburst bombs of the modern era are incredibly clean and radiation is only a risk in a very small area (tens of…

all bombs are bad. nuclear bombs the worst..if you try to argue for them you are hopelessly lost.

Re: AIs can't stop recommending nuclear strikes in war game simulations

#140

Post by jdross »

oceanplexian wrote: post #133

I've spoken with engineers who worked on nuclear weapons systems, the consensus is that the public is deeply misinformed about how they work, the dangers, and the implications of weapons being used. The AI is actually right here. The biggest danger of a nuclear weapon is being hit by flying debris. Fusion airburst bombs of the modern era are incredibly clean and radiation is only a risk in a very small area (tens of…

I think the consensus is the biggest danger of a nuclear weapon being used is that it will result in way more nuclear weapons being used.

The specific damage of a single nuclear weapon is far outweighed by thousands of them hitting population centers in an escalation of force

Post reply on HN