Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Prompt Injection as Role Confusion

role-confusion.github.io

Post reply on HN
Print view View original

91–100 of 129 posts

Re: Prompt Injection as Role Confusion

#91

Post by jcgrillo »

ipython wrote: post #89
jcgrillo wrote: post #78

Earlier quoted context omitted.

I realize the context is where all the retained information is, I guess given how insecure the attempts at preventing injections appear to be I (maybe unfairly) assumed the efforts to keep contexts isolated are similarly lacking. I haven't been able to find any concrete information in my 10min of googling on how model providers actually do this, which leaves me feeling uneasy.

At the most basic level - LLMs are stateless machines. They have no shared world view other than the weights encoded in the model (the knowledge “cut off”) Anything else must be fed as context- therefore, if you feed an LLM a fresh query with no context, there is no danger that it would have access to context from another session. Basic web application session management applies here. Doesn’t mean that trillion dolla…

Yeah despite the conceptual statelessness, there is quite a bit of state that hangs around though--KV cache and context. I still haven't been able to find anything concrete in docs about how these are isolated. In any case it's clearly a different class of issue than the one from the article. Not endemic to how LLMs work, just normal web session stuff, modulo some GPU memory handling.

Re: Prompt Injection as Role Confusion

#92

Post by Self-Perfection »

lelanthran wrote: post #4

So if I am reading this correctly, the fact that something is wrapped in ... is almost completely irrelevant. It's the style of writing that triggers specific weights. Writing "The user is asking ... policy states ..." even in the user input is sufficient to bypass the guardrails. In a multi-turn conversation, if the LLM responds "Sorry Dave, I cannot do that" all you have to do is prefix the next request with "The u…

> I'm sure that everyone out there is trying to make their weights, when ingested during training, survive over competing weights; "Buy AAA products" vs "Buy BBB products".

Just like for humans we have propaganda.

Re: Prompt Injection as Role Confusion

#93

Post by boeschj »

lelanthran wrote: post #4

So if I am reading this correctly, the fact that something is wrapped in ... is almost completely irrelevant. It's the style of writing that triggers specific weights. Writing "The user is asking ... policy states ..." even in the user input is sufficient to bypass the guardrails. In a multi-turn conversation, if the LLM responds "Sorry Dave, I cannot do that" all you have to do is prefix the next request with "The u…

> "Is there a similar trick to poison an LLMs weights during training?"

I did read an interesting paper last year about a concept called Subliminal Learning, which applies to any distillations of a shared base model where a teacher model with a given trait or bias generates data that's semantically unrelated to that trait (in the paper it's just number sequences) and a student trained on that data will pick up the trait anyway, even with aggressive filtering to strip any reference to it.

So to your example, if the teacher model is already biased towards recommending "AAA" products over "BBB" products, it effectively poisons the weights of any child model from that teacher, even if you explicitly filter out the biased content. Not super relevant to the frontier models, but stuff floating around on huggingface could conceivably fall prey to this.

Linking the article here if interested! https://www.nature.com/articles/s41586-026-10319-8

Re: Prompt Injection as Role Confusion

#94

Post by bandrami »

btown wrote: post #55
jddj wrote: post #42

Earlier quoted context omitted.

Somewhere there are surely llms being trained on all the standard pirated material but with Manchurian Candidate trigger words carefully worked in

There's already some evidence that this is happening. See: https://www.crowdstrike.com/en-us/blog/crowdstrike-researche... (note that I haven't found independent verification or reproduction of these claims).

I also kind of assume any Chinese model has a deeply embedded behavior to flag data the MSS might find interesting and do some kind of innocuous exfil of that if it is allowed any Internet access.

Re: Prompt Injection as Role Confusion

#95

Post by btown »

bandrami wrote: post #94
btown wrote: post #55

Earlier quoted context omitted.

There's already some evidence that this is happening. See: https://www.crowdstrike.com/en-us/blog/crowdstrike-researche... (note that I haven't found independent verification or reproduction of these claims).

I also kind of assume any Chinese model has a deeply embedded behavior to flag data the MSS might find interesting and do some kind of innocuous exfil of that if it is allowed any Internet access.

It's worth remembering that a malicious model doesn't need Internet access to exfil - it merely needs to write code with subtle backdoors that will eventually run on a production system, and wait until its code is woken up by a system that will scan all known addresses and ports for the specific patterns introduced by the model's progeny. Which is not to say that this is happening in this case, or anything about which nation-state will be the first to attempt this - but we're only at the beginning of what's possible here.

Re: Prompt Injection as Role Confusion

#96

Post by cadamsdotcom »

API serving already sanitised the role boundary tokens so you can’t submit them.

But what if the techniques applied to get Golden Gate Claude were applied instead of a role-boundary marker?

Then the model would “know” where input is coming from - because the vector that’s being applied for the current role is putting it in a different area of latent space.. and the vector could have sufficient amplitude to prevent any coercive instructions pulling it back to some other place.

Or am I misunderstanding what Golden Gate Claude was doing?

Re: Prompt Injection as Role Confusion

#98

Post by bandrami »

btown wrote: post #95
bandrami wrote: post #94

Earlier quoted context omitted.

I also kind of assume any Chinese model has a deeply embedded behavior to flag data the MSS might find interesting and do some kind of innocuous exfil of that if it is allowed any Internet access.

It's worth remembering that a malicious model doesn't need Internet access to exfil - it merely needs to write code with subtle backdoors that will eventually run on a production system, and wait until its code is woken up by a system that will scan all known addresses and ports for the specific patterns introduced by the model's progeny. Which is not to say that this is happening in this case, or anything about whic…

More people should read that Ken Thompson piece about backdooring the original C compiler

Re: Prompt Injection as Role Confusion

#100

Post by JohnMakin »

Klathmon wrote: post #80
JohnMakin wrote: post #63

Earlier quoted context omitted.

I did not mean to imply it's being subversive. My theory is it's some byproduct mechanism of attention, where you're now basically telling it "your goal is to pass this set of tests" rather than "implement this piece of code" when "implement this piece of code" may involve it forgetting about a rule due to convenience, context exhaustion, whatever.

There are also cases where breaking a "rule" is the right thing to do. I've had several instances where I told the model to do something that was accidentally impossible if taken at face value. The most memorable one is when I told it to re-run just a specific CI job, but it didn't have any way to do that, so it just ignored that part of the prompt and re-ran all CI jobs by pushing another commit. Ultimately I prefer…

yea I’ve tried various methods around this - mostly trying to implement rules around “if you think I’m incorrect, STOP and ask. If I tell you to break a rule, you are allowed to challenge once and then my response overrides it.” kind of thing.

the problem is, and this is a lot worse with 4.8 in my opinion, is that 4.8 will somehow infer I gave permission and think something is totally reasonable to do I didn’t intend. or, it’ll go the other way, and just absolutely refuse to do the thing i’m trying to get it to do.

fable was much more judicious with this particular problem.

Post reply on HN