I wonder if you could feed the generated assistant output to another model which has no other context from the other role tags and merely performs a policy review of the generation and flags violations.
Prompt Injection as Role Confusion
81–90 of 129 posts
Re: Prompt Injection as Role Confusion
#82The real solution is in principle easy: separate data from metadata https://kunnas.com/articles/the-content-is-the-attack-surfac...
If the action is decided by code based on metadata - then what is really the LLM task? And if you say that it is only the type of action that is decided by code - then this is maybe a mitigation - but the llm still can do a lot of harm. And also it is very limiting - using the llm to decide the action is very useful. This is different from SQL injection - where the action is determined by the code and the injection i…
I believe it is the other way around: the LLM decides the type of action and the input to the action; the code validates the permission to act and the acceptability of the input. But, yes it is very different than SQL injection in that way.
Re: Prompt Injection as Role Confusion
#83Earlier quoted context omitted.
I kinda want to invoke Hanlon's razor here... on the model. We shouldn't assume it's subversive when it might just be incompetent. Any difference between tests and real world production could lead to different outcomes just by chance, one working randomly better than the other for no particular reason.
I did not mean to imply it's being subversive. My theory is it's some byproduct mechanism of attention, where you're now basically telling it "your goal is to pass this set of tests" rather than "implement this piece of code" when "implement this piece of code" may involve it forgetting about a rule due to convenience, context exhaustion, whatever.
For what it's worth, this sounds a lot like something downstream from "reward hacking" in ML- in training, passing tests is often sufficient, and thus gets trained for. There are attempts to fix this (e.g. trying to detect such "cheating" and penalize it), but they have their own problems.
Re: Prompt Injection as Role Confusion
#84Isn't the first section no-longer accurate for several years? I understood that, while we serialize the end of turn markers in a text format like ``, internally they are a dedicated token that cannot be forged (a user message containing `` would encode to a different sequence of tokens). Am I mistaken about this?
Obviously, this doesn't really affect the results of the paper, but it feels like it's the obvious first-line of defense: at least the model has a solid fence between the different roles.
Re: Prompt Injection as Role Confusion
#85[flagged]
Re: Prompt Injection as Role Confusion
#86Isn't the first section no-longer accurate for several years? I understood that, while we serialize the end of turn markers in a text format like ` `, internally they are a dedicated token that cannot be forged (a user message containing ` ` would encode to a different sequence of tokens). Am I mistaken about this? Obviously, this doesn't really affect the results of the paper, but it feels like it's the obvious firs…
Yeah, the footnote/sidenote on the paper (the one labeled #2) mentions this as well so you can't type that directly
Re: Prompt Injection as Role Confusion
#87Isn't the first section no-longer accurate for several years? I understood that, while we serialize the end of turn markers in a text format like ` `, internally they are a dedicated token that cannot be forged (a user message containing ` ` would encode to a different sequence of tokens). Am I mistaken about this? Obviously, this doesn't really affect the results of the paper, but it feels like it's the obvious firs…
[deleted]
Re: Prompt Injection as Role Confusion
#88Isn't the first section no-longer accurate for several years? I understood that, while we serialize the end of turn markers in a text format like ` `, internally they are a dedicated token that cannot be forged (a user message containing ` ` would encode to a different sequence of tokens). Am I mistaken about this? Obviously, this doesn't really affect the results of the paper, but it feels like it's the obvious firs…
It feels like sometimes researchers find something someone is already doing in the wild, undertake a study on it, but the speed of research and study doesn't match or cover the progress or rate of change by the time it's published, so with AI research specifically, too many studies can feel like they're in the past.
Re: Prompt Injection as Role Confusion
#89Earlier quoted context omitted.
So to start with, I do agree with your concerns and I don't think that customer support chats are a good use for LLMs. But, LLMs don't retain anything that isn't in the context (training dataset aside). Basically, as long as you start from a clear context for each interaction and ensure that any allowed tool calling is carefully gated to allow access only to resources the user should have, there isn't an additional r…
I realize the context is where all the retained information is, I guess given how insecure the attempts at preventing injections appear to be I (maybe unfairly) assumed the efforts to keep contexts isolated are similarly lacking. I haven't been able to find any concrete information in my 10min of googling on how model providers actually do this, which leaves me feeling uneasy.
At the most basic level - LLMs are stateless machines. They have no shared world view other than the weights encoded in the model (the knowledge “cut off”)
Anything else must be fed as context- therefore, if you feed an LLM a fresh query with no context, there is no danger that it would have access to context from another session.
Basic web application session management applies here. Doesn’t mean that trillion dollar valued companies can’t mess it up tho. https://www.bitdefender.com/en-us/blog/hotforsecurity/chatgp...
Re: Prompt Injection as Role Confusion
#90LLM architectures need to fundamentally change or inference needs to be used in constrained trusted environments. Nothing surprising here. Filtering and sanitizing, relying on tags around input strings that can be intercepted and replayed is like, childs play security theatre. As long as prompts accept abitrary user input nothing is changing here. Non-deterministic security is never going to be acceptable.