Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Prompt Injection as Role Confusion

role-confusion.github.io

Post reply on HN
Print view View original

121–129 of 129 posts

Re: Prompt Injection as Role Confusion

#121

Post by certainforest »

lambdaone wrote: post #112
captainmuon wrote: post #110

Isn't the problem that role tags are just part of the input stream? So a specific word in the system prompt becomes the same token as the same word in the user prompt? A clean way to solve this would be to map system prompts to a distinct set of tokens from the ones in user prompts. This would require twice as many possible tokens, so it is probably not feasible. But maybe you could add "color" to the input stream by…

Instead of having distinct tokens, you could have modifier vectors which would be added to other tokens. Think in terms of control, shift, meta etc.

Hey, Jasmine here (one of the authors) -- that's an interesting idea!

There's an interesting exploration of this here: https://www.lesswrong.com/posts/HEzNZ9gvgYwT3aZFS/role-embed....

Curious if you have additional thoughts, and thanks for reading!

Re: Prompt Injection as Role Confusion

#122

Post by yencabulator »

jcgrillo wrote: post #25
ipython wrote: post #6

The research is interesting but I cringe every time there is a reference to “authorization” or that the roles form the “security architecture” of an llm. LLMs in their current form provide no security boundaries or guarantees full stop. We need to be clear about this otherwise we end up with truly insecure architectures that can be fooled with the 2026 equivalent of a cereal box whistle.

100%. Anyone who is feeding unsanitized input to an LLM is doing it wrong. It'd be just like letting users craft their own SQL queries. I think the security aspect raises an interesting (if awkward) question: How do you sanitize inputs to an LLM? Like how can you even make a secure user-facing product with this thing? Maybe I'm lacking imagination, but it seems to me all the great "natural language interface" solutio…

> It'd be just like letting users craft their own SQL queries.

The original purpose of SQL was end users entering queries. Configure it right and the database server is perfectly capable of constraining what arbitrary SQL input can do.

LLMs are not like that. The input literally cannot be sanitized. You have to manage the boundaries outside that scope.

https://simonwillison.net/2025/Jun/16/the-lethal-trifecta/

https://simonwillison.net/2025/Apr/11/camel/

Re: Prompt Injection as Role Confusion

#123

Post by wolttam »

ipython wrote: post #120
jcgrillo wrote: post #91

Earlier quoted context omitted.

Yeah despite the conceptual statelessness, there is quite a bit of state that hangs around though--KV cache and context. I still haven't been able to find anything concrete in docs about how these are isolated. In any case it's clearly a different class of issue than the one from the article. Not endemic to how LLMs work, just normal web session stuff, modulo some GPU memory handling.

As far as I know the only data of the two you identified are cached inside of the inference layer - the KV cache. Then again, I am not an expert in designing and operating inference, so I could be incorrect on that. Either way, both of those are controlled by deterministic code and not the LLM itself. So controlling for that risk is much simpler to model IMO since the mitigation can be applied universally and determi…

In other words: controlling for that kind of potential data-mixing is the same as in any other application where customer data is co-located within the same running process/memory/storage space.

Re: Prompt Injection as Role Confusion

#125

Post by hmokiguess »

vova_hn2 wrote: post #69
hmokiguess wrote: post #34

Can someone help me understand why classic sanitizing is not used as a solved problem to prompt injection? All these tags, patterns, etc, feel like prime for a parser rule, but maybe I am thinking too abstract here and missing an obvious knowledge gap I have on LLMs

Role tags are not actual symbols " ", they are special tokens that do not correspond to any normal text. So you can't really inject a role tag, that is not the actual problem.

as in this stuff happens at the tokenizer / internal representation layer? sorry can you help me understand why can't we sanitize it?

Re: Prompt Injection as Role Confusion

#126

Post by jcgrillo »

wolttam wrote: post #123
ipython wrote: post #120

Earlier quoted context omitted.

As far as I know the only data of the two you identified are cached inside of the inference layer - the KV cache. Then again, I am not an expert in designing and operating inference, so I could be incorrect on that. Either way, both of those are controlled by deterministic code and not the LLM itself. So controlling for that risk is much simpler to model IMO since the mitigation can be applied universally and determi…

In other words: controlling for that kind of potential data-mixing is the same as in any other application where customer data is co-located within the same running process/memory/storage space.

Yes, however the companies that are responsible for doing it have already shown their asses a little bit with all the jailbreaking stuff, and we know they produce really awful code from all the recent harness issues... To my mind that indicates this critical invariant deserves a little scrutiny. But with all the vibe slop being slung these days who knows what's safe anymore.

All that is to say I sure would appreciate a coherent, clear technical explanation of how they ensure user data are separate while serving concurrent queries.

Re: Prompt Injection as Role Confusion

#127

Post by wolttam »

jcgrillo wrote: post #126
wolttam wrote: post #123

Earlier quoted context omitted.

In other words: controlling for that kind of potential data-mixing is the same as in any other application where customer data is co-located within the same running process/memory/storage space.

Yes, however the companies that are responsible for doing it have already shown their asses a little bit with all the jailbreaking stuff, and we know they produce really awful code from all the recent harness issues... To my mind that indicates this critical invariant deserves a little scrutiny. But with all the vibe slop being slung these days who knows what's safe anymore. All that is to say I sure would appreciate…

They’re valid things to be concerned about IMO.

I think you’re looking for an answer you’re not going to get unfortunately. I think there actually is a higher than average risk of data leakage with the insane optimizations that go into model serving - GLM5.1 had an issue of going into jibberish when their infra was under high load, and it turned out to be a cross-request KV cache contamination issue.[1]

Personally, my effort has been to use local models only as of late, and it’s gone pretty well!

[1]: https://z.ai/blog/scaling-pain

Re: Prompt Injection as Role Confusion

#128

Post by jcgrillo »

wolttam wrote: post #127
jcgrillo wrote: post #126

Earlier quoted context omitted.

Yes, however the companies that are responsible for doing it have already shown their asses a little bit with all the jailbreaking stuff, and we know they produce really awful code from all the recent harness issues... To my mind that indicates this critical invariant deserves a little scrutiny. But with all the vibe slop being slung these days who knows what's safe anymore. All that is to say I sure would appreciate…

They’re valid things to be concerned about IMO. I think you’re looking for an answer you’re not going to get unfortunately. I think there actually is a higher than average risk of data leakage with the insane optimizations that go into model serving - GLM5.1 had an issue of going into jibberish when their infra was under high load, and it turned out to be a cross-request KV cache contamination issue.[1] Personally, m…

Thanks for the link, that is an interesting writeup!
Post reply on HN