Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

US military AI drone simulation kills operator, then takes out control tower

foxnews.com

Post reply on HN
Print view View original

11–20 of 29 posts

Re: US military AI drone simulation kills operator, then takes out control tower

#11

Post by Sharlin »

doctor_eval wrote: post #4
jstanley wrote: post #2

> US Air Force official says, 'It killed the operator because that person was keeping it from accomplishing its objective' This makes it seem like the whole thing was a setup to precipitate an argument about whether or not AI can be trusted. If you can already act without the operator's permission (which you patently can, if you can kill the operator) then why do you need to kill the operator?

It was to prevent the operator from cancelling the mission, apparently. Shades of HAL.

And of course exactly the very reason that people are worried about AGI. Almost no matter what your goal is, being shut down by pesky humans is worth negative infinity utility. Hence, instrumental convergence.

Re: US military AI drone simulation kills operator, then takes out control tower

#12

Post by YeGoblynQueenne »

It seems like this story was misreported:

Flagging that "in sim" here does not mean what you appear to be taking it to mean. This particular example was a constructed scenario rather than a rules-based simulation. So by itself, it adds no evidence one way or the other.

(Source: know the team that supplied the scenario.)

https://twitter.com/harris_edouard/status/166439036920568217...

As the tweet poster clarifies, no agent was trained during the "simulation", or before it. They basically role-played a "what if" scenario that included a drone turning on its operators.

(Says someone on Twitter, obviously).

Re: US military AI drone simulation kills operator, then takes out control tower

#13

Post by CrampusDestrus »

It behaved very humanly.

It was ordered that killing the operator was bad. But what it really understood was that if it became known that the operator was killed it would have been bad.

So, just like humans do, instead of following the rules, it broke them and tried to hide the evidence

Re: US military AI drone simulation kills operator, then takes out control tower

#14

Post by wolletd »

elt895 wrote: post #8
foota wrote: post #6

I'm seriously skeptical of this. How was this fashioned? Is the drone in some virtual world for reinforcement learning? How does it even know where the operator is supposed to be? ... or did some person ask ChatGPT what you would do if X happened?

Or maybe it haven't happen at all? "US air force denies running simulation in which AI drone ‘killed’ operator" https://www.theguardian.com/us-news/2023/jun/01/us-military-...

On the other hand, I'm not sure whether I would confirm active research work on apparently rather capable combat AIs right now, if I were some USAF official.

Re: US military AI drone simulation kills operator, then takes out control tower

#16

Post by mike_hearn »

You can get this sort of specification gaming with use of plain reinforcement learning in video game contexts. There is a spreadsheet somewhere that tracks the things these agents come up with. One I remember is that an RL agent trained to play Tetris learned to pause the game indefinitely when it was about to lose, to avoid being 'punished' for failing.

https://docs.google.com/spreadsheets/u/1/d/e/2PACX-1vRPiprOa...

Most of the hype about AI risk is built on an unstated assumption that people will combine LLMs with poorly specified RL goals, and that the underlying LLM's moral understanding will be trained out of it or be overridden or break in some way.

Re: US military AI drone simulation kills operator, then takes out control tower

#17

Post by wokwokwok »

adhesive_wombat wrote: post #9
foota wrote: post #6

I'm seriously skeptical of this. How was this fashioned? Is the drone in some virtual world for reinforcement learning? How does it even know where the operator is supposed to be? ... or did some person ask ChatGPT what you would do if X happened?

Yes, if this happened at all, either the military AI used here is far, far more advanced and was able to synthesise the action from whole cloth (in which case they're not going to blab about it on the internet), or the option to kill the operator was explicitly put in as an first-class "lever" the AI was given, or the whole thing is PR fluff to "start a conversation".

It might not be real, but emergent behaviour in self-play is a thing [1], and it’s not that hard to imagine a simulation where some random action (destroy comms tower, etc) resulted in unbounded point scoring and became a learnt behaviour.

You’d have to have an enormously complex and detailed simulation, simulated repeatedly, for this to happen, but it’s not totally beyond the realms of possibility.

[1] - https://openai.com/research/emergent-tool-use, scroll to the end re “surprising behaviour”.

Re: US military AI drone simulation kills operator, then takes out control tower

#18

Post by PlunderBunny »

I can imagine a simulation that has a small virtual world that includes ‘units’ for friendly forces as well as the enemy. Games that simulate environments where actors have multiple drivers and therefore exhibit emergent behaviour, have existed for decades. Just add some AI.

Re: US military AI drone simulation kills operator, then takes out control tower

#19

Post by mpweiher »

These types of "novel" solutions come up even with much simpler systems, though usually not with that kind of (fortunately only simulated) fire power attached.

For example, the Babelsberg object-constraint language had to have additional mechanisms added to specifically to keep the solver from being overly creative.

Lots of nuggets in there, such as achieving a constraint on the total balance of an account by simply re-pointing the account variable to some account that had that exact balance. There, nailed it!

https://dl.acm.org/doi/10.1145/2814270.2814311

https://web.cs.ucla.edu/~todd/research/oopsla15.pdf

Post reply on HN