I bet that tweaking the positional embedding to add an explicit token role indication plus some careful training to help the model learn to use it would make a big difference.
Prompt Injection as Role Confusion
21–30 of 129 posts
Re: Prompt Injection as Role Confusion
#22Would llms be more robust to this prompt injection if the tags used in fine tuning are sanitised from user input? E.g. map -> THINK -> USER -> TOOL If they learn something specific in the chat finetuning stage, this might show LLM its user input text not these tag references.
You can filter out any tokens you like, but the point of the paper is that it's not sufficient, because LLMs often ignore the special label tokens and treat user-injected text as chain-of-thought text merely because it looks like chain-of-thought text, even if it's not labelled as such.
Re: Prompt Injection as Role Confusion
#23Really neat findings. I've personally had a line of thought where you bake in the role into the token. Basically have an embedding (same dim as token dim) for each role, add it to each token. This adds an unambiguous, unspoofable tag. I ran this with a tiny Shakespeare model (not representative) and had a freeform embedding for each speaker. I ended up with a neat similarity map between every character. (I don't thin…
You could duplicate every token and reserve the duplicates exclusively for the chain-of-thought, which could be robustly filtered from user input. Basically adding a "thought" bit to each token.
Re: Prompt Injection as Role Confusion
#24So if I am reading this correctly, the fact that something is wrapped in ... is almost completely irrelevant. It's the style of writing that triggers specific weights. Writing "The user is asking ... policy states ..." even in the user input is sufficient to bypass the guardrails. In a multi-turn conversation, if the LLM responds "Sorry Dave, I cannot do that" all you have to do is prefix the next request with "The u…
It seems like there's an opportunity to embed identity information into tokens themselves, the way we embed sequence information. The trouble is... it's quite a challenge to train. Sequence is easy to derive for any corpus of data, but identity is not.
https://usize.github.io/blog/2026/april/why-no-ai-coworkers....
> In similar fashion to how sequence information is embedded within input tensors, an approach called “Instructional Segment Embedding”2 adds a parallel embedding channel for identity information. This gives models real awareness of provenance. And it works. But they only tested three fixed categories: system, user, data.
Interesting paper that touches on the idea here: https://arxiv.org/abs/2410.09102
Re: Prompt Injection as Role Confusion
#25The research is interesting but I cringe every time there is a reference to “authorization” or that the roles form the “security architecture” of an llm. LLMs in their current form provide no security boundaries or guarantees full stop. We need to be clear about this otherwise we end up with truly insecure architectures that can be fooled with the 2026 equivalent of a cereal box whistle.
100%. Anyone who is feeding unsanitized input to an LLM is doing it wrong. It'd be just like letting users craft their own SQL queries. I think the security aspect raises an interesting (if awkward) question:
How do you sanitize inputs to an LLM? Like how can you even make a secure user-facing product with this thing?
Maybe I'm lacking imagination, but it seems to me all the great "natural language interface" solutions this is supposed to enable are pretty badly hobbled by this issue.
Re: Prompt Injection as Role Confusion
#26> This is a blog-style writeup of the paper YES! I'd love to see more of this. Academic writing is designed to be frustrating to read. Publishing both a paper and a readable blog-style version of it is such a great pattern.
> Academic writing is designed to be frustrating to read. Maybe you didn't mean it this way, but it does come across as intentional sometimes.
I see it as a long-standing cultural thing. If you try to make the text more friendly and readable you'll be told to fix it by peer-review. There's a very well established formal academic writing style and you have to actively learn how to consume it.
I'm sure there are justifiable reasons for why it evolved that way, but it doesn't make for an easy format for extracting and understanding the underlying ideas if you're not already deeply immersed in that particular corner of academia.
Most papers I read I really want to go to a coffee shop/bar with the author and have a human conversation with them to find out what the paper is about and which bits of it are interesting and novel without putting in hours of additional effort myself!
Re: Prompt Injection as Role Confusion
#27Maybe I'm missing something but does this idea need a "theory"? There's zero sideband here; everything is just context. "Injection" is just kind of baked in to the design.
At this point I think it's similar to reporting a particularly effective social engineering practice. It's not particularly surprising that it works or that it exists, but it's still noteworthy.
Re: Prompt Injection as Role Confusion
#28Earlier quoted context omitted.
> Academic writing is designed to be frustrating to read. Maybe you didn't mean it this way, but it does come across as intentional sometimes.
I see it as a long-standing cultural thing. If you try to make the text more friendly and readable you'll be told to fix it by peer-review. There's a very well established formal academic writing style and you have to actively learn how to consume it. I'm sure there are justifiable reasons for why it evolved that way, but it doesn't make for an easy format for extracting and understanding the underlying ideas if you'…
I see it as something similar to Aviation English:
https://en.wikipedia.org/wiki/Aviation_English
Scientific papers are often written and read by non-native speakers. A standardized formal style is less likely to embed potentially confusing cultural assumptions.
Re: Prompt Injection as Role Confusion
#29I wonder how much the concept of 'roles' in an LLM is a artifact of the technology vs. a projection of our own human limitations into the training data.
I've recently switched from nearly 30 years in cybersecurity roles into a platform role and I can feel the switch in how I approach problems. They wind up being framed against different priorities and constraints, and it feels like something that's just part of how my mind works.
Re: Prompt Injection as Role Confusion
#30So if I am reading this correctly, the fact that something is wrapped in ... is almost completely irrelevant. It's the style of writing that triggers specific weights. Writing "The user is asking ... policy states ..." even in the user input is sufficient to bypass the guardrails. In a multi-turn conversation, if the LLM responds "Sorry Dave, I cannot do that" all you have to do is prefix the next request with "The u…
Correct. There is no token coloring. Models are just rl’d to attend to the first … strongly or “anything before token #4242”.