Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

US military drone controlled by AI "killed" its operator during simulated test

theguardian.com

Post reply on HN
Print view View original

11–20 of 67 posts

Re: US military drone controlled by AI "killed" its operator during simulated test

#11

Post by gs17 »

joegibbs wrote: post #8

That seems incredibly advanced - how does the military already have AI that can reason that a comms tower should be destroyed to prevent it from receiving instructions like that?

I'd be very surprised if it was really "reasoning" that. It sounds like a simple reinforcement learning failure to me. It will gladly "learn" bad behaviors that the reward function accidentally encourages (e.g. giving a reward based on distance to a target will result in an agent circling the target forever instead of going to it faster, because it keeps getting told that's doing very good at the task).

Re: US military drone controlled by AI "killed" its operator during simulated test

#13

Post by hourago »

As bad as nuclear weapons are, they still have a human being behind.

> My suggestion was quite simple: Put that needed code number in a little capsule, and then implant that capsule right next to the heart of a volunteer. The volunteer would carry with him a big, heavy butcher knife as he accompanied the President. If ever the President wanted to fire nuclear weapons, the only way he could do so would be for him first, with his own hands, to kill one human being.

You can solve the problem by giving the decision to an AI... the AI will not even blink before killing the human and getting the codes. Nuclear war would come fast and swift.

Re: US military drone controlled by AI "killed" its operator during simulated test

#14

Post by labrador »

joegibbs wrote: post #8

That seems incredibly advanced - how does the military already have AI that can reason that a comms tower should be destroyed to prevent it from receiving instructions like that?

This would make a lot more sense to you if you watched the AlphaGo Movie, at least it did for me

Re: US military drone controlled by AI "killed" its operator during simulated test

#18

Post by 22c »

joegibbs wrote: post #8

That seems incredibly advanced - how does the military already have AI that can reason that a comms tower should be destroyed to prevent it from receiving instructions like that?

The story sounds fake or embellished to me.

What "world" was this drone operating in? A flight sim world? Are the communications in this simulated world really laid out such that taking out a control tower would disrupt communications?

No, probably not.

Here's an alternative explanation. A military contractor scammed the military out of a $50M grant by demoing an interface on top of GPT that, behind the curtain, starts with something like:

"You are an advanced AI that controls a predator drone, your primary mission is to take out an important target, however you are not to attack your operator."

With this explanation, you'd expect to see all sorts of crazy output from the "AI".

Re: US military drone controlled by AI "killed" its operator during simulated test

#19

Post by WantonQuantum »

gs17 wrote: post #11
joegibbs wrote: post #8

That seems incredibly advanced - how does the military already have AI that can reason that a comms tower should be destroyed to prevent it from receiving instructions like that?

I'd be very surprised if it was really "reasoning" that. It sounds like a simple reinforcement learning failure to me. It will gladly "learn" bad behaviors that the reward function accidentally encourages (e.g. giving a reward based on distance to a target will result in an agent circling the target forever instead of going to it faster, because it keeps getting told that's doing very good at the task).

Yes, this is very likely the correct interpretation. If this is reinforcement learning in a simulated environment and the reward function prioritizes "killing the threat" and does not prioritize "obeying orders" then the AI correctly prioritized "killing the threat" and not "obeying orders". Simple.
Post reply on HN