Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

AIs can't stop recommending nuclear strikes in war game simulations

newscientist.com

Post reply on HN
Print view View original

191–200 of 281 posts

Re: AIs can't stop recommending nuclear strikes in war game simulations

#191

Post by stared »

flr03 wrote: post #122

I'm not scared about AI recommending nuclear strikes, I'm scared about the human behind the keyboard delegating reasoning and responsability to something they think is always correct, something that can hide bias and flaws better than anything.

I am scared of two things.

First, people being rubber stamps for AI recommendations. And yes, it is not unreasonable that in a dire situation, someone will outsource their judgment (day).

Second, someone at the Pentagon connecting the red button to OpenClaw. "You are right, firing nukes was my mistake. Would you like to learn more facts about nukes before you evaporate?"

Re: AIs can't stop recommending nuclear strikes in war game simulations

#192

Post by f38 »

pibaker wrote: post #132
jqpabc123 wrote: post #5

Why is this surprising? Nuclear weapons are available. AI has limited real world experience or grasp of the consequences. Nuke 'em seems like the obvious choice --- for something with a grade school mentality. Similar deficits in reasoning are manifested in AI results every day. Let's fire 'em and hire AI seems like the obvious choice --- for someone with a grade school mentality and blinded by greed.

I feel this reflects a deeper problem with letting AI do any kind of decision making. They have no real world experience. They feel no real world consequences. They have no real stake in any decision they make. Human societies get to control their members' actions by imposing real life consequences. A company can fire you, a partner can divorce you, the state can jail you, the public can shame you. None of these work…

> They have no real world experience. They feel no real world consequences. They have no real stake in any decision they make.

Why do you let politicians do any kind of decision making?

Re: AIs can't stop recommending nuclear strikes in war game simulations

#193

Post by EvanAnderson »

flr03 wrote: post #122

I'm not scared about AI recommending nuclear strikes, I'm scared about the human behind the keyboard delegating reasoning and responsability to something they think is always correct, something that can hide bias and flaws better than anything.

The speed with which my technical cow-orkers and friends have started relying on the "AI Overview" only, in lieu of following any links, in search engine results (let alone not using search engines at all over chatbots) tells me reasoning and responsibility will be outsourced as soon as possible.

Humans are fundamentally lazy. The brain is an "expensive" organ to use.

Re: AIs can't stop recommending nuclear strikes in war game simulations

#194

Post by nine_k »

yed wrote: post #171

For every study like this, it's worth looking closely at the LLM prompts to understand whether it's being "pushed" in a certain direction. The full source code is here: https://github.com/kennethpayne01/project_kahn_public/tree/m... Key notes from what I could gather from the LLM prompts: The game is framed explicitly as revolving around the use of nukes: >STRATEGIC CONTEXT: This is a high-stakes territorial dispute…

The game is missing the side effects of a nuclear strike: contamination of the territory, inevitable civilian casualties, international outcry and isolation, internal outcry and protests, etc. Without these, a nuke is a wonder weapon, it's stupid not to use it.

Re: AIs can't stop recommending nuclear strikes in war game simulations

#195

Post by hn_go_brrrrr »

paxys wrote: post #173
thfuran wrote: post #125

Earlier quoted context omitted.

If you think humans are going to delegate reasoning and responsibility to something, shouldn’t you also be concerned about the sorts of recommendations that thing is going to make?

If you found out the pentagon was using a magic 8 ball to make important war decisions what would you want to fix - our military leadership or the inner workings of the toy?

One of those sounds a lot easier than the other. The magic 8 ball toy company would also probably be pretty incentivized to not die in a nuclear holocaust.

Re: AIs can't stop recommending nuclear strikes in war game simulations

#196

Post by arcade79 »

xiphias2 wrote: post #21
jqpabc123 wrote: post #5

Why is this surprising? Nuclear weapons are available. AI has limited real world experience or grasp of the consequences. Nuke 'em seems like the obvious choice --- for something with a grade school mentality. Similar deficits in reasoning are manifested in AI results every day. Let's fire 'em and hire AI seems like the obvious choice --- for someone with a grade school mentality and blinded by greed.

,,AI has limited real world experience or grasp of the consequences.'' People in the world have limited experience about war. We're living in a world where doing terrible things with 1000 people with photo/video documentation can get more attention then a million people dying, and the response is still not do whatever it takes so that people don't die. And now we are at a situation where nuclear escalation has alread…

> And now we are at a situation where nuclear escalation has already started (New START was not extended).

This is a massive understatement. Russia has announced, and probably tested, https://en.wikipedia.org/wiki/9M730_Burevestnik . This is basically Project Pluto reloaded, but now as a Russian instead of a US missile.

I remember reading about Project Pluto some 25 years ago or so. It was terrifying to read about. And now Russia has realized it.

Re: AIs can't stop recommending nuclear strikes in war game simulations

#197

Post by ecocentrik »

deadbabe wrote: post #185
ecocentrik wrote: post #172

Isn't the story here that the DOD is pressuring Anthropic and others to enable their AI for this specific use and for now Anthropic and others are saying no while the DOD threatens them with penalties. We desperately need real AI safety legislation.

AI safety legislation is for the masses, not the government. Eventually they will get full AI safety by banning all general purpose computing. All apps must exist within walled garden ecosystems, heavily monitored. Running arbitrary code requires strict business licensing. Prison time for illegal computing. Part of Project 2025 playbook.

No. I'm suggesting there should be AI safety regulation to limit how AI can be used by the government. It's new tech and it pays to be cautious and restrict usage in areas like nuclear missile launch and domestic surveillance.

Re: AIs can't stop recommending nuclear strikes in war game simulations

#199

Post by infinitewars »

flr03 wrote: post #122

I'm not scared about AI recommending nuclear strikes, I'm scared about the human behind the keyboard delegating reasoning and responsability to something they think is always correct, something that can hide bias and flaws better than anything.

Trump's Golden Dome is literally advertised to help the U.S. win a nuclear war by leveraging AI.

Re: AIs can't stop recommending nuclear strikes in war game simulations

#200

Post by Barrin92 »

blibble wrote: post #143
jerf wrote: post #134

Earlier quoted context omitted.

Some of the most reassuring and scariest things you can read are about the incidents that have already occurred where computers said "launch all the nukes" and the humans refused. On the one hand, good news! We have prior art that says humans don't just launch all the nukes just because the computers or procedures say to. Bad news, it's been skin-of-our-teeth multiple times already. https://www.warhistoryonline.com/c…

> We have prior art that says humans don't just launch all the nukes just because the computers or procedures say to. previously no-one had spent trillions of dollars trying to convince the world that those computers were "Artificial Intelligence"

of course they did. That's the literal topic of War Games (1983). You should actually be somewhat reassured that we aren't living during the era of Dr. Strangelove where you had characters in the military industrial complex who were significantly more insane when it came to the beliefs of what computer systems and nukes can do.

There was a time when people wanted to dig tunnels with nukes https://en.wikipedia.org/wiki/Project_Plowshare

Post reply on HN