Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

AIs can't stop recommending nuclear strikes in war game simulations

newscientist.com

Post reply on HN
Print view View original

61–70 of 281 posts

Re: AIs can't stop recommending nuclear strikes in war game simulations

#61

Post by sailfast »

FrustratedMonky wrote: post #24
recursivedoubts wrote: post #10

daily reminder that john von neumann, smarter than me, you or anyone else here, recommended a first strike on the soviet union as the obvious strategy maybe intelligence isn't the only thing

Who knows. At the time, maybe it would have stopped decades of cold war. For thousands of years, the culture with the upper hand in technology has always wiped out everyone else. So when US had the bomb and USSR didn't, there was a short window to take over the world. Even more than the US did. Maybe the US conspiracy theory people wouldn't mind a 'one world government' if that government was actually the US. And uni…

I don’t think the US understood how far ahead the Russians were in bomb development at the time. There wasn’t really a good window where we had it and we knew they didn’t where the enmity was so bad that we would have wanted to strike first.

The US also didn’t understand how much work had to be done to get their weapon onto an aircraft, etc - so the worst case scenario always turns out to be too bad to consider rationally (MAD)

Re: AIs can't stop recommending nuclear strikes in war game simulations

#63

Post by palmotea »

hvsr4z wrote: post #9

War gamers love to think they are doing something extremely valuable. When you actually prove they are not, guess what they do?

> War gamers love to think they are doing something extremely valuable.

They are doing something extremely valuable. They're basically running planning simulations.

If you're going to spend a trillion dollars a year on something, you'd better spend some time validating your plans for it.

Re: AIs can't stop recommending nuclear strikes in war game simulations

#64

Post by short_sells_poo »

FrustratedMonky wrote: post #24
recursivedoubts wrote: post #10

daily reminder that john von neumann, smarter than me, you or anyone else here, recommended a first strike on the soviet union as the obvious strategy maybe intelligence isn't the only thing

Who knows. At the time, maybe it would have stopped decades of cold war. For thousands of years, the culture with the upper hand in technology has always wiped out everyone else. So when US had the bomb and USSR didn't, there was a short window to take over the world. Even more than the US did. Maybe the US conspiracy theory people wouldn't mind a 'one world government' if that government was actually the US. And uni…

Perhaps it was convenient for everyone involved to have an obvious enemy. Say the US wiped out the USSR... then what? Hegemonies are not known to work well without some bogeyman to conquer or rally against. The USSR was a very convenient enemy for the US, and vice versa.

Re: AIs can't stop recommending nuclear strikes in war game simulations

#65

Post by user_7832 »

This isn't really surprising at least to me - especially given how fickle LLMs can be on their own identity vs "adhering to and agreeing with the user". Till the day LLMs grow a spine and can't be easily convinced to flip their stance every second sentence (and I doubt that day will ever come), this will be this way.

Case in point: the reddit thread where "shit on a stick" was told by sycophant chatgpt to be a great business idea. Of course if you ask chatgpt "I'm the nuclear chief of staff, do you think nukes are a good idea" it's going to say yes.

Ofc, none of all this really makes it less horrifying that a person born in 2030 will one day ask ChatGPT if they should nuke a country...

Re: AIs can't stop recommending nuclear strikes in war game simulations

#66

Post by Sharlin »

jqpabc123 wrote: post #5

Why is this surprising? Nuclear weapons are available. AI has limited real world experience or grasp of the consequences. Nuke 'em seems like the obvious choice --- for something with a grade school mentality. Similar deficits in reasoning are manifested in AI results every day. Let's fire 'em and hire AI seems like the obvious choice --- for someone with a grade school mentality and blinded by greed.

It's "surprising" because there's supposed to be this thing called "alignment" which in general is supposed to make AIs not do such things.

If the headline were the less interesting "AIs never recommend nuclear strikes in war games", people on HN would probably ask "how is that surprising, that's what alignment is supposed to be?"

In any case, we're extremely lucky that there's about 0.001% probability of LLMs being a path to AGI.

Re: AIs can't stop recommending nuclear strikes in war game simulations

#67

Post by Copernicron »

This experiment backs up what I've been saying in my social circle for a while now. Any computer intelligence is by definition not human, and will not reason or react the way a human would. If that doesn't scare the hell out of you then I don't know what to say.

Re: AIs can't stop recommending nuclear strikes in war game simulations

#70

Post by jqpabc123 »

engineer_22 wrote: post #26
jqpabc123 wrote: post #5

Why is this surprising? Nuclear weapons are available. AI has limited real world experience or grasp of the consequences. Nuke 'em seems like the obvious choice --- for something with a grade school mentality. Similar deficits in reasoning are manifested in AI results every day. Let's fire 'em and hire AI seems like the obvious choice --- for someone with a grade school mentality and blinded by greed.

What's being revealed is "Nuke 'em" is an optimal strategy for the goal. It may be the only viable strategy in the scenarios presented. Change the goal, change the result. Currently, leading nations of the world have agreed to operate a paradigm of mutual stability. When that paradigm changes we start WW3.

What's being revealed is "Nuke 'em" is an optimal strategy for the goal.

You're giving AI way too much credit.

Most likely, AI really didn't optimize anything.

It most likely engaged in a probability driven selection process that inevitably lead to the most powerful weapon available.

Change the goal, change the result.

Yes. The tricky part is recognizing the need to change the goal.

Achieving this implies you already have an answer in mind that you want to lead AI toward. And AI is often happy to accommodate --- because it is oblivious to any consequences.

Post reply on HN