> until some enemy combatant uses prompt injection to get a drone to bomb a different target.
You got me interested in how Palantir is using an LLM. From Palantir's demo [1]:
> In the video demo above, a military operator tasked with monitoring the Eastern European theater discovers enemy forces massing near the border and responds by asking a ChatGPT-style digital assistant for help with deploying reconnaissance drones, ginning up tactical responses to the perceived aggression and even organize the jamming of the enemy's communications. The AIP is shown helping estimate the enemy's composition and capabilities by launching a Reaper drone on a reconnaissance mission in response the to operator's request for better pictures, and suggesting appropriate responses given the discovery of an armored element.
Where the LLM operates is at the command and control level, from what I can tell effectively running a combat operations center which is usually a field level officers job.
If LLMs are limited to giving high level instructions on rote tasks, that's a pretty good job for it. Thankfully, things like strikes require at least three layers of observation and approval with each layer getting a denying vote. I think if the military is going to use technology like this it's going to put an even greater emphasis on the control frameworks we use in theater.
That said, there's very little error margin when you're talking full scale theater combat. For instance, if you deploy HIMARS to an area that has aviation active you'll likely take down aircraft upon the HIMARS reentry from orbit due to the pressure change. Another could be overreliance on technological markers like Blue Force Trackers (BFTs); troop misidentification does still occur. You'd need a human at every authorizing layer is my point, and maybe more importantly a human that does not innately trust the output of the machine.
Last, and maybe my more nuanced thought is that too much information is also damaging in theater. Misdirection occurs quite a bit by troops in contact; understandably so if you're being shot at and being chased building to building while clearing backlayed ordinance your bearings are likely a bit off. One of the functions of the COC Commander is to executively silence some inputs and put more assets on more directly observing the troops in contact. LLMs would need to get incredibly good at not just rote operations but interpreting new challenges, some which have probably never been seen or recorded before in order to be even remotely viable.
1: https://www.engadget.com/palantir-shows-off-an-ai-that-can-g...