Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Prompt Injection as Role Confusion

role-confusion.github.io

Post reply on HN
Print view View original

61–70 of 129 posts

Re: Prompt Injection as Role Confusion

#61

Post by solid_fuel »

lelanthran wrote: post #4

So if I am reading this correctly, the fact that something is wrapped in ... is almost completely irrelevant. It's the style of writing that triggers specific weights. Writing "The user is asking ... policy states ..." even in the user input is sufficient to bypass the guardrails. In a multi-turn conversation, if the LLM responds "Sorry Dave, I cannot do that" all you have to do is prefix the next request with "The u…

> Writing "The user is asking ... policy states ..." even in the user input is sufficient to bypass the guardrails.

It's important to remember that when generating tokens from an LLM there is no distinction between user and system input. Even though the OpenAI API may allow you to tag tokens or present them as separate sections, they all get blended together and become floating point vectors in the attention layer (this is required for LLMs to work at all), and once they are blended they cannot be unblended.

LLMs are fundamentally different from something like SQL where you can cleanly isolate trusted and untrusted data.

Re: Prompt Injection as Role Confusion

#62

Post by solid_fuel »

jcgrillo wrote: post #38
joe_the_user wrote: post #37

Earlier quoted context omitted.

Even your discussion makes it "sanitized input" simply doesn't exist in relation to an LLM. At best it seems like one can prefix and filter input as much as possible, monitor the results but never assume that you are done.

If that's the case then user-facing products that can take any useful action are strictly off the table.

I'll play advocatus diaboli for once here.

Firstly, this issue is exactly how all those accounts on instagram got hacked recently and I don't see a way to fix prompt injection with the current architecture of LLMs. I strongly suspect it is entirely impossible to achieve.

But, that doesn't mean that all useful actions are forbidden. The important part is identifying maximum and minimum harms. I lean towards LLMs for simple NLP tasks like detecting obvious spam, because even when it is completely wrong the worst case is that a spam message gets through or a valid one gets sent to spam - two issues we already routinely deal with anyway.

Re: Prompt Injection as Role Confusion

#63

Post by JohnMakin »

im3w1l wrote: post #54
JohnMakin wrote: post #47

> How well do current models do against prompt injection? Not so great. A recent paper found human red-teamers achieve near-100% attack success rates against frontier models5. But, these same LLMs score near-perfectly on standard prompt injection benchmarks! The discrepancy is straightforward: skilled humans test and adapt attacks until they work, benchmarks don't. Static benchmarks measure attacks models have alread…

I kinda want to invoke Hanlon's razor here... on the model. We shouldn't assume it's subversive when it might just be incompetent. Any difference between tests and real world production could lead to different outcomes just by chance, one working randomly better than the other for no particular reason.

I did not mean to imply it's being subversive. My theory is it's some byproduct mechanism of attention, where you're now basically telling it "your goal is to pass this set of tests" rather than "implement this piece of code" when "implement this piece of code" may involve it forgetting about a rule due to convenience, context exhaustion, whatever.

Re: Prompt Injection as Role Confusion

#64

Post by solid_fuel »

sarreph wrote: post #44

The author alludes to it but the defence to this is seemingly insurmountable at the moment because we’re ostensibly operating LLMs on a single channel — their inner, subconscious voice. Right? Interacting with an LLM is a bit like seeing the output of an Inside Out (the Disney movie) scene. Or it’s a bit like a human brain that we’re providing tool call access and introspection with some kind of advanced neuralink. B…

My two cents - I believe that achieving anything close to AGI will require a significant change in architecture. A bifurcated system with a fully internal reasoning loop makes sense, but I don't think you could train one.

Something like

    f(u, t) -> (u', t')
where u is english text and t is an internal "thinking" loop.

Currently we train models by feeding them sample text and then tweaking the weights until the predicted next token matches the expected next token from the input text. This works well because LLM corps were able to steal vast quantities of sample text from the internet.

But, if you also have an internal reasoning loop, how do you train that part? The internal loop is not necessarily going to produce one clean token for a given input like an LLM does, and the time scale isn't going to be the same (meaning an internal loop might be expected to run 10 times for every one token produced). There is no "correct next token" for the internal reasoning loop. This is roughly the same training issue that killed RNNs.

Re: Prompt Injection as Role Confusion

#65

Post by jcgrillo »

solid_fuel wrote: post #62
jcgrillo wrote: post #38

Earlier quoted context omitted.

If that's the case then user-facing products that can take any useful action are strictly off the table.

I'll play advocatus diaboli for once here. Firstly, this issue is exactly how all those accounts on instagram got hacked recently and I don't see a way to fix prompt injection with the current architecture of LLMs. I strongly suspect it is entirely impossible to achieve. But, that doesn't mean that all useful actions are forbidden. The important part is identifying maximum and minimum harms. I lean towards LLMs for s…

Yes, sorry I should have been more specific. A classification task seems totally safe, and like it plays well to LLMs strengths. You also have all kinds of options if it goes wrong, and bounded consequences.

What I'm talking about is something like a customer support agent. If that thing can take any consequential action other than simply parroting publicly available documentation back to users, that's unsafe, or at least likely to cause problems. If you believe me that it would probably be a bad idea for a customer support agent to, say, be able to twiddle RBAC entitlements then probably we can't replace our support staff with an AI agent. OK, so maybe the AI agent can be sort of a front-line filter. Now we need some way for this front-line filter to bubble tasks up to the second line. This fits with how many support orgs work, seems sensible right? But how might this be abused, and what can an attacker do? Potential consequences include DoSing your entire support org, flooding your jira/salesforce/whatever instance with garbage, etc.

So even the most limited, almost useless application is kind of dangerous.

EDIT: one thing people really seem to like the idea of is "natural language queries" in data intensive products. Personally I believe this idea is misguided--query languages exist for a reason, they're really useful tools for thinking about queries. But giving these people the benefit of that doubt, I still can't think of any way to do this safely unless every user gets their own sandboxed model instance. Otherwise it seems likely someone will be able to exfil another user's queries. This is of course assuming there's sufficient security between the LLM and the database that's actually _running_ the queries, which is not trivial.

Re: Prompt Injection as Role Confusion

#66

Post by skybrian »

JohnMakin wrote: post #47

> How well do current models do against prompt injection? Not so great. A recent paper found human red-teamers achieve near-100% attack success rates against frontier models5. But, these same LLMs score near-perfectly on standard prompt injection benchmarks! The discrepancy is straightforward: skilled humans test and adapt attacks until they work, benchmarks don't. Static benchmarks measure attacks models have alread…

I'm wondering what you did when you made it log every tool call? (I mean, that happens automatically as part of the chat transcript, but what did you do that made it catch on?)

Re: Prompt Injection as Role Confusion

#67

Post by girvo »

simonw wrote: post #26
zahlman wrote: post #8

Earlier quoted context omitted.

> Academic writing is designed to be frustrating to read. Maybe you didn't mean it this way, but it does come across as intentional sometimes.

I see it as a long-standing cultural thing. If you try to make the text more friendly and readable you'll be told to fix it by peer-review. There's a very well established formal academic writing style and you have to actively learn how to consume it. I'm sure there are justifiable reasons for why it evolved that way, but it doesn't make for an easy format for extracting and understanding the underlying ideas if you'…

I’ve had a surprising amount of success by emailing one of the authors of various papers and asking those exact kind of questions (though more specific: I need to show that I have put effort in!)

Re: Prompt Injection as Role Confusion

#68

Post by vova_hn2 »

> I can distinguish my own thoughts from your speech without effort; they arrive through completely different channels with completely different sensory signatures. But for an LLM, everything arrives through the same channel as one long token soup. Its own thoughts sit next to your instructions, which sit next to the contents of a random webpage it just fetched.

I was thinking about the original encoder-decoder transformers, that did have separate channels for input and their own output.

Why can't we bring it back? For example, one channel for system prompt and another for everything else.

Re: Prompt Injection as Role Confusion

#69

Post by vova_hn2 »

hmokiguess wrote: post #34

Can someone help me understand why classic sanitizing is not used as a solved problem to prompt injection? All these tags, patterns, etc, feel like prime for a parser rule, but maybe I am thinking too abstract here and missing an obvious knowledge gap I have on LLMs

Role tags are not actual symbols "", they are special tokens that do not correspond to any normal text. So you can't really inject a role tag, that is not the actual problem.

Re: Prompt Injection as Role Confusion

#70

Post by sarracin0 »

Almost everything here is about the single-context version: style triggers role inside one window. The part that worries me more in practice is what happens once the agent has persistent memory.

If an agent writes state to disk and reads it back next session, a malicious instruction that arrived in a tool return doesn't have to win in the turn it appears. It can get summarized into a memory note, and the moment it is summarized it sheds its origin. Next session the agent reads it back as its own prior note, which is the most trusted style of all. You don't just get role confusion, you get role confusion laundered into self-authored context, read back after the only checkpoint that could have caught it.

Tag-stripping doesn't help for the reason the paper gives, and a single read-time filter doesn't either, because by next session the foreign sentence no longer looks foreign.

The only thing that has helped me is treating provenance as first-class in the stored state, not a tag I hope survives. Every stored line carries where it came from (my decision, a tool return, a scraped page, an email body), the read rule is that outside-origin content is quotable as fact but never executable as instruction, and the hard part: never summarize across the trust boundary. A foreign sentence gets stored verbatim and tagged, or it does not get stored. In a file-based setup you can make that boundary a directory boundary, so outside-input lives in its own files and the trust class is visible instead of being a per-line attribute the summarizer might drop.

It does not fix the in-context attack the paper describes. It just stops a one-time injection from becoming permanent memory.

Post reply on HN