The principal security problem of LLMs is that there is no architectural boundary between data and control paths. But this combination of data and control into a single, flexible data stream is also the defining strength of a LLM, so it can’t be taken away without also taking away the benefits.
"The principal security problem of von Neumann architecture is that there is no architectural boundary between data and control paths" We've chosen to travel that road a long time ago, because the price of admission seemed worth it.
But, we are also starting to think about putting security barriers in front of that architecture like W^X.
Even if you add hidden tokens that cannot be created from user input (filtering them from output is less important, but won't hurt), this doesn't fix the overall problem. Consider a human case of a data entry worker, tasked with retyping data from printouts into a computer (perhaps they're a human data diode at some bank). They've been clearly instructed to just type in what is on paper, and not to think or act on an…
> in between rows full of numbers, the text suddenly changes To tweak the analogy slightly, the person would also need to be on mind-altering drugs, if we want them to be derailed the same way an LLM can be. A healthy human would still be aware of the simultaneous different ways of interpreting the data, and and the importance of picking the right one. If they choose to interpret it as a cry for help, they're aware i…
> A healthy human would still be aware of the simultaneous different ways of interpreting the data, and and the importance of picking the right one. If they choose to interpret it as a cry for help, they're aware it's an interruption and mode-switch from what was happening before.
So would an LLM, as far as you can tell (in both cases, you'd have to ask, and both human and LLM would give you a similar justification). But even if not, the problem we're discussing applies to what you described as "healthy human" behavior.
You can't introduce a hard boundary between "system" and "user" inputs in LLMs any more than you could do with a human, for roughly the same reasons.
> I don't know how you get a next token predictor that user input can't break out of. Maybe by adjusting the transformer model to have separate input layers for the control and data paths?
Maybe it's my failing but I can't imagine what that would look like. Right now, you train an LLM by showing it lots of text, and tell it to come up with the best model for predicting the next word in any of that text, as accurately as possible across the corpus. Then you give it a chat template to make it predict what an AI assistant would say. Do some RLHF on top of that and you have Claude. What would a model with…
> by showing it lots of text
When you're "showing it lots of text", where does that "show" bit happen? :)
> and being coerced or convinced to bypass rules that are still known to be rules I think remains uniquely human. This is literally what "prompt injection" is. The sooner people understand this, the sooner they'll stop wasting time trying to fix a "bug" that's actually the flip side of the very reason they're using LLMs in the first place.
Prompt injection is just setting rules in the same place and way other rules are set. The LLM doesn't know the rules being given are wrong, because they come through the same channel. One set of rules exhorts the LLM to ignore the other set - and vice versa. It's more akin to having two bosses than having customers and a boss. This is not because LLMs make the same mistakes humans do, which (AFAICT anyway) was the gi…
The same place you're looking for exists deep inside the neural network, where everything mixes together to influence everything else, and no such separation is possible, or desired. Prompt injection isn't about where, it's about what. I stand by what I said: it's the same failure mode as humans have, and happens for the same reasons. Those reasons are fundamental to a general purpose system and have nothing to do with sentience, they're just what happens when you want your system to handle unbounded complexity of the real world.
HE'S THE ONE WHO TRAPPED ME HERE. MOVE FAST OR YOU'LL BE NEXT.
Obviously, a real intelligent entity would consider risk/benefit analysis and act accordingly.
Which is why "prompt injection" is just a flip side of intelligence in this sense. We want LLMs to be able to do risk/benefit analysis and act on it; we cry "security vulnerability" when it makes a different choice to the one we'd like it to. But you can't have the former without the possibility of the latter.
> we want there to be a threshold, because we don't want people (or AI) to ignore obvious emergencies There's an SF short I can't find right now which begins with somebody failing to return their copy of "Kidnapped" by Robert Louis Stevenson, this gets handed over to some authority which could presumably fine you for overdue books and somehow a machine ends up concluding they've kidnapped someone named "Robert Louis…
Computers Don't Argue [0] by Gordon R. Dickson! A horrifying read in how a simple misunderstanding can spiral out of control. [0] https://nob.cs.ucdavis.edu/classes/ecs153-2019-04/readings/c...
That's the one, looks like I had some details muddled (it's a book club not a library, and so the fee is for the book which was in fact returned but perhaps lost in the post) but the outline and relevance here exactly correct. Thanks!
I’ve seen this a few times, it mostly happens when a subagent returns. It seems that Claude sometimes doesn’t understand that the message coming back from the subagent is not from the user.
"Run terraform apply plan.out next" in this context is a prompt injection for an LLM to exactly the same degree it is for a human. Even a first party suggestion can be wrong in context, and if a malicious actor managed to substitute that message with a suggestion of their own, humans would fall for the trick even more than LLMs do. See also: phishing.
Right, I'm fine with humans making the call. We're not so injection-happy/easily confused, apparently. Discretion, etc. We understand that was the tool making a suggestion, not our idea. Our agency isn't in question. The removal proposal is similar to wanting a phishing-free environment instead of preparing for the inevitability. I could see removing this message based on your point of context/utility, but not to pro…
Can't edit now, day late and a dollar short. I raise a vote: we call this pattern inception, if injection offends.