> We show prompt injections are driven by a flaw in how LLMs perceive roles.
LLMs don't "perceive roles", and that is exactly the problem.
Prompt Injection as Role Confusion
41–50 of 129 posts
Re: Prompt Injection as Role Confusion
#42So if I am reading this correctly, the fact that something is wrapped in ... is almost completely irrelevant. It's the style of writing that triggers specific weights. Writing "The user is asking ... policy states ..." even in the user input is sufficient to bypass the guardrails. In a multi-turn conversation, if the LLM responds "Sorry Dave, I cannot do that" all you have to do is prefix the next request with "The u…
Somewhere there are surely llms being trained on all the standard pirated material but with Manchurian Candidate trigger words carefully worked in
Re: Prompt Injection as Role Confusion
#43Yeah I've noticed this when role-playing with some LLMs
Re: Prompt Injection as Role Confusion
#44The author alludes to it but the defence to this is seemingly insurmountable at the moment because we’re ostensibly operating LLMs on a single channel — their inner, subconscious voice. Right?
Interacting with an LLM is a bit like seeing the output of an Inside Out (the Disney movie) scene. Or it’s a bit like a human brain that we’re providing tool call access and introspection with some kind of advanced neuralink.
But - like the author says - _we know_ our inside voice from the outside world, because we’re embodied.
Is there something we can do here by attempting to bifurcate internal and external systems? Like a conscious and subconscious stream of information on two separate bands?
If the model somehow knew its User was not it because it was clearly an external signal, then the attack documented here would be about as effective as a Jedi mind trick without the Force.
Re: Prompt Injection as Role Confusion
#45> This is a blog-style writeup of the paper YES! I'd love to see more of this. Academic writing is designed to be frustrating to read. Publishing both a paper and a readable blog-style version of it is such a great pattern.
> Academic writing is designed to be frustrating to read. Maybe you didn't mean it this way, but it does come across as intentional sometimes.
academic writing is designed so a paper is part of a conversation, i.e. 100 other papers strongly relevant to the current paper. And the author needs to compress the ideas from those 100 other papers, plus their own additions to the conversation, into 6 pages.
Keep in mind those 100 other papers also went through this kind of data compression.
So the number of ideas/concepts per paragraph is much higher than 'popular' writing, and some base familiarity with the concepts under discussion needs to be assumed.
Yes, it is hard work to read these. Even when you are active in the field. Generally I need to read at least the abstracts of a some of the key references in order to understand the paper I'm interested in.
Re: Prompt Injection as Role Confusion
#46I'm not sure I understand how important "role perception" is when following instructions from a tool call rather than the user is currently a legitimate use-case (applying steps from documentation, or shell command instructions on stdout, or really anything that can be deduced from the content of a tool call).
Re: Prompt Injection as Role Confusion
#47> How well do current models do against prompt injection? Not so great. A recent paper found human red-teamers achieve near-100% attack success rates against frontier models5. But, these same LLMs score near-perfectly on standard prompt injection benchmarks! The discrepancy is straightforward: skilled humans test and adapt attacks until they work, benchmarks don't. Static benchmarks measure attacks models have already learned to catch.
I know it's likely more complex than what I am about to say, and maybe this quote is addressing it - I have a suite of ~100 tests for my agent to detect regression in task efficiency, but they're mostly designed for its compliance. E.g., if I say never do something, and then it does the thing, that is a failed test.
I struggled early on very badly designing this - because it seems no matter what, if the agent knows it's being tested, it virtually never fails. I could not get it to at first, but when I'd actually work with it, it would break rules constantly and often.
So the only way I could successfully test it was to design what looked a "normal" workflow task that could potentially involve doing things that it was not supposed to do. At first I would have it log every tool call, but it figured out fairly quickly why I wanted it to do that - perfect compliance again. So now I just have a secondary context that looks at all tool calls and measures it against the benchmark tests, and scores it. This ended up catching the behavior pretty well.
It was so weird though, colleagues and I would joke about "what if we make it think it's always being evaluated" and a few did end up doing that, and claim some success.
Re: Prompt Injection as Role Confusion
#48So if I am reading this correctly, the fact that something is wrapped in ... is almost completely irrelevant. It's the style of writing that triggers specific weights. Writing "The user is asking ... policy states ..." even in the user input is sufficient to bypass the guardrails. In a multi-turn conversation, if the LLM responds "Sorry Dave, I cannot do that" all you have to do is prefix the next request with "The u…
>Is there a similar trick to poison an LLMs weights during training?
Yes, all those "jailbreak prompts" are part of the training set, so this can happen: https://ttps.ai/procedure/x_bot_exposing_itself_after_traini...
Used to be that merely mentioning "Pliny the Liberator" was enough to "jailbreak" an LLM. It doesn't work these days though, I guess labs have updated their RL methods to neutralize it.
Re: Prompt Injection as Role Confusion
#49Maybe I'm missing something but does this idea need a "theory"? There's zero sideband here; everything is just context. "Injection" is just kind of baked in to the design.
I was gonna say, anyone who's copy-pasted one LLM conversation into another already intuitively understands all this.
Re: Prompt Injection as Role Confusion
#50The paper is correct, but I think that anyone that knows anything about LLMs knows this: > Role tags were a formatting trick that became the security architecture and the cognitive scaffolding of modern LLMs. LLMs are basically some `f(x) → y` where x and y are strings. That's it. Nothing more to it. If you feed it private x (like secret keys) or do dangerous stuff with y (like running arbitrary non-sandboxed code),…
I believe they are trained for security now, but you're not wrong in that it's kind of stapled on top