Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

US military drone controlled by AI "killed" its operator during simulated test

theguardian.com

Post reply on HN
Print view View original

31–40 of 67 posts

Re: US military drone controlled by AI "killed" its operator during simulated test

#31

Post by t-3 »

hardware2win wrote: post #27
arisAlexis wrote: post #22

Earlier quoted context omitted.

Really? You think we can use AI for free to code all our work and write all our essays but the army doesn't have something that reasons who to kill?

Id believe it because it isnt 1970 or something like that anymore, private comp sci should be ahead

Military/intel/fedgov are still the main drivers of technology, actually probably moreso now than ever. The low hanging fruit are fewer and fewer and giant monopolies on the scale of Bell Labs don't really exist anymore. Just take a look at In-Q-Tel for a public example, then think about the academic research sponsored directly and indirectly, then there's all the nonpublic stuff on top. A ton of companies exist only to supply government, and more depend on government as a/the major customer. The cutting edge will always be there, following the money.

Re: US military drone controlled by AI "killed" its operator during simulated test

#32

Post by silisili »

hourago wrote: post #13

As bad as nuclear weapons are, they still have a human being behind. > My suggestion was quite simple: Put that needed code number in a little capsule, and then implant that capsule right next to the heart of a volunteer. The volunteer would carry with him a big, heavy butcher knife as he accompanied the President. If ever the President wanted to fire nuclear weapons, the only way he could do so would be for him firs…

That's my entire problem with AI or any machine decision. It has zero emotion, zero empathy, zero care.

Imagine if AI were running the subs in the Cold War. None of us would probably even be here. We may be stupid creatures, but we're still creatures...

https://en.m.wikipedia.org/wiki/Vasily_Arkhipov

Re: US military drone controlled by AI "killed" its operator during simulated test

#33

Post by IEnjoyTapNinja »

The Guardian hasn't done any due diligence and is spreading fake news based on an interviewer who seems to have misunderstood the context of the exercise.

"He notes that one simulated test saw an AI-enabled drone tasked with a SEAD mission to identify and destroy SAM sites, with the final go/no go given by the human."

It is being established in the beginning of the story, that the drone needed confirmation from a human operator to attack a target, but no explanation is given how the drone would be able to kill its operator without his confirmation.

This is obviously absurd.

What I believe happened in reality: this was not a simulation, but a scenario. Meaning a story written to test the behavior of soldiers in certain situations. The drone did not behave according to the decisions taken by an AI model, but according to the decisions taken by a human instructor, who was trying to get the trainees to think outside the box.

Re: US military drone controlled by AI "killed" its operator during simulated test

#34

Post by arisAlexis »

utucuro wrote: post #29
arisAlexis wrote: post #26

Earlier quoted context omitted.

Read up about the "alignment problem". It is one that ai researchers cannot Crack and its literally what you wrote. The idea that we could never in any way define goals align with us.

I already know about the efforts about AI alignment - this is the reason I'm afraid about natural stupidity. I really do not want to see the alignment issue solved in my lifetime.

Cringe comment: the reason why will not see it both you and me will be the same if it doesn't get solved :)

Re: US military drone controlled by AI "killed" its operator during simulated test

#35

Post by arrowsmith »

silisili wrote: post #32
hourago wrote: post #13

As bad as nuclear weapons are, they still have a human being behind. > My suggestion was quite simple: Put that needed code number in a little capsule, and then implant that capsule right next to the heart of a volunteer. The volunteer would carry with him a big, heavy butcher knife as he accompanied the President. If ever the President wanted to fire nuclear weapons, the only way he could do so would be for him firs…

That's my entire problem with AI or any machine decision. It has zero emotion, zero empathy, zero care. Imagine if AI were running the subs in the Cold War. None of us would probably even be here. We may be stupid creatures, but we're still creatures... https://en.m.wikipedia.org/wiki/Vasily_Arkhipov

> It has zero emotion, zero empathy, zero care.

So do some humans.

Re: US military drone controlled by AI "killed" its operator during simulated test

#36

Post by Jon_Lowtek »

IEnjoyTapNinja wrote: post #33

The Guardian hasn't done any due diligence and is spreading fake news based on an interviewer who seems to have misunderstood the context of the exercise. "He notes that one simulated test saw an AI-enabled drone tasked with a SEAD mission to identify and destroy SAM sites, with the final go/no go given by the human." It is being established in the beginning of the story, that the drone needed confirmation from a hum…

The Guardian is citing aerosociety.com and has a link to this source, which uses direct quotes from the Chief of AI Test and Operations, USAF, at the future combat air space capabilities summit.

> What I believe happened in reality

You can believe what you want, but that is not what Hamilton said had happened.

Re: US military drone controlled by AI "killed" its operator during simulated test

#37

Post by vivegi »

What is the role of the AI and what is the distinct role of the operator?

It looks like this is a principal agent problem (https://en.wikipedia.org/wiki/Principal%E2%80%93agent_proble...):

  The principal–agent problem refers to the conflict in interests and priorities that arises when one person or entity takes actions on behalf of another person or entity.
The same issues occur with self-driving cars where it is expected that the driver take over from the automation anytime (eg: driver wants to stop but AI wants to go or vice versa).

Re: US military drone controlled by AI "killed" its operator during simulated test

#38

Post by fennecfoxy »

Definitely just bad model/test conditions/scoring design. Of course the military is using home-grown fisher price models.

The reward function should primarily be based on following the continued instructions of the handler, not taking the first instruction and then following it to the letter.

What's funny though, is that the model proved that it was adept at the task they gave it. Trying to kill the operator, then when adjusted pivoting to destroying the comms tower the operator used. That's still clever.

As per usual the problem isn't the tool, it's the tool using the tool. Set proper goals and train the model properly and it would work perfectly. I think weapons should always require a human in the loop, but the problem is that there'll be an arms-race where some countries (you know who) will ignore these principles and build fully autonomous no-human weapons.

Then, when our systems can't react fast enough to defend ourselves because they need a human in the loop, what will we do? Throw out our principles and engage in fully-autonomous weaponry as well? It's the nuclear weapons problem all over again...

Re: US military drone controlled by AI "killed" its operator during simulated test

#40

Post by fennecfoxy »

hourago wrote: post #13

As bad as nuclear weapons are, they still have a human being behind. > My suggestion was quite simple: Put that needed code number in a little capsule, and then implant that capsule right next to the heart of a volunteer. The volunteer would carry with him a big, heavy butcher knife as he accompanied the President. If ever the President wanted to fire nuclear weapons, the only way he could do so would be for him firs…

Do they, though?

Didn't Russia have the dead man's hand system where any detected launches would result in a completely automatic response?

But in the reality of an arm's race, I guess the core point is that we either: respond to autonomous weapons with our own and break our principle of having a human in the loop, or accept that our offence/defence with a human in the loop is not fast enough to keep up with an enemy's fully autonomous systems, thereby just completely giving up.

Just as with nuclear weapons, the unfortunate outcome isn't "well they're bad so we don't want them" but instead it's "we don't want these, but we have to out of necessity".

What if Russia was the only country with nuclear weapons and all other countries held steadfast principles of not building any nuclear weapons? Pretty obvious outcomes to be honest.

Post reply on HN