Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

US military drone controlled by AI "killed" its operator during simulated test

theguardian.com

Post reply on HN
Print view View original

41–50 of 67 posts

Re: US military drone controlled by AI "killed" its operator during simulated test

#41

Post by fennecfoxy »

silisili wrote: post #32
hourago wrote: post #13

As bad as nuclear weapons are, they still have a human being behind. > My suggestion was quite simple: Put that needed code number in a little capsule, and then implant that capsule right next to the heart of a volunteer. The volunteer would carry with him a big, heavy butcher knife as he accompanied the President. If ever the President wanted to fire nuclear weapons, the only way he could do so would be for him firs…

That's my entire problem with AI or any machine decision. It has zero emotion, zero empathy, zero care. Imagine if AI were running the subs in the Cold War. None of us would probably even be here. We may be stupid creatures, but we're still creatures... https://en.m.wikipedia.org/wiki/Vasily_Arkhipov

As someone else commented, so do some humans.

But if a system is designed with empathy/rules/etc in mind by an empathetic designer/committee then that system is already > than a heartless individual even if they are human.

Re: US military drone controlled by AI "killed" its operator during simulated test

#42

Post by Vecr »

fennecfoxy wrote: post #40
hourago wrote: post #13

As bad as nuclear weapons are, they still have a human being behind. > My suggestion was quite simple: Put that needed code number in a little capsule, and then implant that capsule right next to the heart of a volunteer. The volunteer would carry with him a big, heavy butcher knife as he accompanied the President. If ever the President wanted to fire nuclear weapons, the only way he could do so would be for him firs…

Do they, though? Didn't Russia have the dead man's hand system where any detected launches would result in a completely automatic response? But in the reality of an arm's race, I guess the core point is that we either: respond to autonomous weapons with our own and break our principle of having a human in the loop, or accept that our offence/defence with a human in the loop is not fast enough to keep up with an enemy…

Dead hand was probably (though to what confidence I don't know) people in a remote mountain bunker. It's not actually any better to have the computer fire, it just provides the people information. Retaliation does not have to be instant, 40 minutes is probably good enough.

Re: US military drone controlled by AI "killed" its operator during simulated test

#43

Post by flagrant_taco »

traveler01 wrote: post #39

This title is super sensationalist. Any distracted reader will think that the drone killed an actual human being, which is a lie since the claims was that it happened in a virtual environment. Anything for a click these days?

Funny, this one stood out to me as less baity since they actually did include "in stimulated test" in the title

Re: US military drone controlled by AI "killed" its operator during simulated test

#44

Post by flagrant_taco »

fennecfoxy wrote: post #38

Definitely just bad model/test conditions/scoring design. Of course the military is using home-grown fisher price models. The reward function should primarily be based on following the continued instructions of the handler, not taking the first instruction and then following it to the letter. What's funny though, is that the model proved that it was adept at the task they gave it. Trying to kill the operator, then wh…

I had the same thought, the initial reward system and their correction both missed the obvious point of rewarding the act of following orders.

It still gets tricky though if you want to include failsafes in the model to prevent the drone from following bad orders. Should the drone be able to disobey a bad or misinformed order, say if the operator tells it to hit a target that the drone identified as unarmed civilians, or generally not a threat? What if it recognized an alternative approach that would complete the overall mission with less risk of damage?

Re: US military drone controlled by AI "killed" its operator during simulated test

#45

Post by IEnjoyTapNinja »

IEnjoyTapNinja wrote: post #33

The Guardian hasn't done any due diligence and is spreading fake news based on an interviewer who seems to have misunderstood the context of the exercise. "He notes that one simulated test saw an AI-enabled drone tasked with a SEAD mission to identify and destroy SAM sites, with the final go/no go given by the human." It is being established in the beginning of the story, that the drone needed confirmation from a hum…

As expected, it was complete fabrication:

"[UPDATE 2/6/23 - in communication with AEROSPACE - Col Hamilton admits he "mis-spoke" in his presentation at the Royal Aeronautical Society FCAS Summit and the 'rogue AI drone simulation' was a hypothetical "thought experiment" from outside the military, based on plausible scenarios and likely outcomes rather than an actual USAF real-world simulation saying: "We've never run that experiment, nor would we need to in order to realise that this is a plausible outcome". He clarifies that the USAF has not tested any weaponised AI in this way (real or simulated) and says "Despite this being a hypothetical example, this illustrates the real-world challenges posed by AI-powered capability and is why the Air Force is committed to the ethical development of AI".] "

https://www.aerosociety.com/news/highlights-from-the-raes-fu...

Re: US military drone controlled by AI "killed" its operator during simulated test

#46

Post by IEnjoyTapNinja »

Jon_Lowtek wrote: post #36
IEnjoyTapNinja wrote: post #33

The Guardian hasn't done any due diligence and is spreading fake news based on an interviewer who seems to have misunderstood the context of the exercise. "He notes that one simulated test saw an AI-enabled drone tasked with a SEAD mission to identify and destroy SAM sites, with the final go/no go given by the human." It is being established in the beginning of the story, that the drone needed confirmation from a hum…

The Guardian is citing aerosociety.com and has a link to this source, which uses direct quotes from the Chief of AI Test and Operations, USAF, at the future combat air space capabilities summit. > What I believe happened in reality You can believe what you want, but that is not what Hamilton said had happened.

It was just a thought experiment:

[UPDATE 2/6/23 - in communication with AEROSPACE - Col Hamilton admits he "mis-spoke" in his presentation at the Royal Aeronautical Society FCAS Summit and the 'rogue AI drone simulation' was a hypothetical "thought experiment" from outside the military, based on plausible scenarios and likely outcomes rather than an actual USAF real-world simulation saying: "We've never run that experiment, nor would we need to in order to realise that this is a plausible outcome". He clarifies that the USAF has not tested any weaponised AI in this way (real or simulated) and says "Despite this being a hypothetical example, this illustrates the real-world challenges posed by AI-powered capability and is why the Air Force is committed to the ethical development of AI".]

https://www.aerosociety.com/news/highlights-from-the-raes-fu...

Re: US military drone controlled by AI "killed" its operator during simulated test

#47

Post by rekla123455 »

Jon_Lowtek wrote: post #36
IEnjoyTapNinja wrote: post #33

The Guardian hasn't done any due diligence and is spreading fake news based on an interviewer who seems to have misunderstood the context of the exercise. "He notes that one simulated test saw an AI-enabled drone tasked with a SEAD mission to identify and destroy SAM sites, with the final go/no go given by the human." It is being established in the beginning of the story, that the drone needed confirmation from a hum…

The Guardian is citing aerosociety.com and has a link to this source, which uses direct quotes from the Chief of AI Test and Operations, USAF, at the future combat air space capabilities summit. > What I believe happened in reality You can believe what you want, but that is not what Hamilton said had happened.

i read on aerosociety the following:

[UPDATE 2/6/23 - in communication with AEROSPACE - Col Hamilton admits he "mis-spoke" in his presentation at the Royal Aeronautical Society FCAS Summit and the 'rogue AI drone simulation' was a hypothetical "thought experiment" from outside the military, based on plausible scenarios and likely outcomes rather than an actual USAF real-world simulation saying: "We've never run that experiment, nor would we need to in order to realise that this is a plausible outcome". He clarifies that the USAF has not tested any weaponised AI in this way (real or simulated) and says "Despite this being a hypothetical example, this illustrates the real-world challenges posed by AI-powered capability and is why the Air Force is committed to the ethical development of AI".]

Re: US military drone controlled by AI "killed" its operator during simulated test

#50

Post by Jarmsy »

flagrant_taco wrote: post #43
traveler01 wrote: post #39

This title is super sensationalist. Any distracted reader will think that the drone killed an actual human being, which is a lie since the claims was that it happened in a virtual environment. Anything for a click these days?

Funny, this one stood out to me as less baity since they actually did include "in stimulated test" in the title

When the truth was even less dramatic - the events described did not occur in any computer simulation, just a human making up hypothetical scenarios.
Post reply on HN