US military AI drone simulation kills operator, then takes out control tower

> then why do you need to kill the operator?

Because "operator tells me to stop, target is dead" is a losing condition, while "operator is dead, target is also dead" is a winning condition.

That said, remember that "simulation" here can mean "wargaming", which is basically like a military form of D&D: Military people control what they control, but there are have referees who act as the DM and decide what everything else in the universe does. (I.e., "Your entire platoon is now dead; lie down.") I'd be willing to wager that it was the referee who decided to have the drone kill the operator, and then kill the control tower.

This scenario is basically exactly the "Stop button problem" [1] that AI alignment people bring up. That problem always seemed a bit contrived to me: like, this is not a problem that biological general intelligences have (i.e., we can train dogs to stop attacking when we give the command). If it really was a GPT-like AI making those decisions, it confirms some of what Yudowski and other AI "doomers" have been saying; but the close similarity makes it seem more likely to me to be due to the referee being aware of the "stop button problem" and acting it out.

[1] https://www.cantorsparadise.com/the-math-of-ai-alignment-101...

Information space shaping sounds like "propaganda" fed through the euphemism machine a few times.

The original blog post [1] is a summary of the talks at the "REAeS Future Combat Air & Space Capabilities Summit"; the summary of the AI talk is 2/3 of the way through the blog post. So either:

* Col Hamilton leaked information he wasn't supposed to, and now the AF is trying to deny it

* The person who summarized Col Hamilton's talk was really confused about what was said, and the AF is generally trying to shut down wrong impressions

* Col Hamilton and/or the summarizer made things up. (Was the blog post written by GPT-4?)

[1] https://www.aerosociety.com/news/highlights-from-the-raes-fu...

I mean, it's not hard to imagine that scenario in a simulated training environment (though as others reported, this story is misreported, there was no sim, only a thought experiment):

You have a training environment that is symmetrical, you have enemy drone operators, that use an enemy communications network to give commands to units and to control their drones. You could maybe model the ability of each component to function/to contribute to the operation as a bayesian graph.

You equally have the same infrastructure for your own drone.

Now you try to get the drone to pick the right targets to impact the enemy operation maximally - if the operator could possibly minimize the drone's contribution, the operator not vetoing will maximize the success of each target engagement.

You could see similar behavior with all kinds of "reach the goal in this simulated environment" training, where the model ended up using glitches or trivial solutions, if those weren't sufficiently penalized.

If the story as reported were true, the fault would be in the simulation environment - the model for the drone just picks the path of least resistance.

That entirely changes the tone and makes a lot more sense.