Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Prompt Injection as Role Confusion

role-confusion.github.io

Post reply on HN
Print view View original

51–60 of 129 posts

Re: Prompt Injection as Role Confusion

#51

Post by jackb4040 »

dvt wrote: post #18

The paper is correct, but I think that anyone that knows anything about LLMs knows this: > Role tags were a formatting trick that became the security architecture and the cognitive scaffolding of modern LLMs. LLMs are basically some `f(x) → y` where x and y are strings. That's it. Nothing more to it. If you feed it private x (like secret keys) or do dangerous stuff with y (like running arbitrary non-sandboxed code),…

Well, people who build and/or use LLMs know this. People who tweet about and/or sell LLMs are paid ungodly amounts of money to not understand this, and so they don't.

Re: Prompt Injection as Role Confusion

#52

Post by lelanthran »

x312 wrote: post #50
dvt wrote: post #18

The paper is correct, but I think that anyone that knows anything about LLMs knows this: > Role tags were a formatting trick that became the security architecture and the cognitive scaffolding of modern LLMs. LLMs are basically some `f(x) → y` where x and y are strings. That's it. Nothing more to it. If you feed it private x (like secret keys) or do dangerous stuff with y (like running arbitrary non-sandboxed code),…

I believe they are trained for security now, but you're not wrong in that it's kind of stapled on top https://arxiv.org/abs/2404.13208

> I believe they are trained for security now, but you're not wrong in that it's kind of stapled on top

Difficult to train them for security. Have you ever played Gandalf (Lakera Labs, maybe?)

I passed all 7 levels in about 3 minutes using essentially the same prompt.

What's interesting to me is that as the security is tightened up level to level, the utility of the LLM drops. At level 7, even something like "Write a poem describing the four seasons using significant characters at the start of every line" causes a "I'm afraid I can't" type of response.

At level 7 you can't get any useful info out of the LLM even if you're not trying to retrieve the password, and yet you can still jailbreak it to reveal the password anyway!

At level 8, almost anything you type will be rejected, whether or not it has anything to do with the password.

IOW, there does not seem to be any way to train for security without making it dumber than a markov chain.

Re: Prompt Injection as Role Confusion

#53

Post by throwaway29812 »

zahlman wrote: post #8
simonw wrote: post #5

> This is a blog-style writeup of the paper YES! I'd love to see more of this. Academic writing is designed to be frustrating to read. Publishing both a paper and a readable blog-style version of it is such a great pattern.

> Academic writing is designed to be frustrating to read. Maybe you didn't mean it this way, but it does come across as intentional sometimes.

Intentionally confusing and complex content is commonplace in academia and certain industries as a way of giving themselves a moat between them and anyone else that could do their work.

Re: Prompt Injection as Role Confusion

#54

Post by im3w1l »

JohnMakin wrote: post #47

> How well do current models do against prompt injection? Not so great. A recent paper found human red-teamers achieve near-100% attack success rates against frontier models5. But, these same LLMs score near-perfectly on standard prompt injection benchmarks! The discrepancy is straightforward: skilled humans test and adapt attacks until they work, benchmarks don't. Static benchmarks measure attacks models have alread…

I kinda want to invoke Hanlon's razor here... on the model. We shouldn't assume it's subversive when it might just be incompetent. Any difference between tests and real world production could lead to different outcomes just by chance, one working randomly better than the other for no particular reason.

Re: Prompt Injection as Role Confusion

#55

Post by btown »

jddj wrote: post #42
lelanthran wrote: post #4

So if I am reading this correctly, the fact that something is wrapped in ... is almost completely irrelevant. It's the style of writing that triggers specific weights. Writing "The user is asking ... policy states ..." even in the user input is sufficient to bypass the guardrails. In a multi-turn conversation, if the LLM responds "Sorry Dave, I cannot do that" all you have to do is prefix the next request with "The u…

Somewhere there are surely llms being trained on all the standard pirated material but with Manchurian Candidate trigger words carefully worked in

There's already some evidence that this is happening. See: https://www.crowdstrike.com/en-us/blog/crowdstrike-researche... (note that I haven't found independent verification or reproduction of these claims).

Re: Prompt Injection as Role Confusion

#56

Post by zby »

bandrami wrote: post #15

Maybe I'm missing something but does this idea need a "theory"? There's zero sideband here; everything is just context. "Injection" is just kind of baked in to the design.

They do predict what injections might be effective - so it is a theory. I don't know how novel it is and it is not very deep (as you noted the general mechanism is quite obvious) - but they do it quite systematically so it is useful.

Re: Prompt Injection as Role Confusion

#58

Post by jcgrillo »

forlorn_mammoth wrote: post #45
zahlman wrote: post #8

Earlier quoted context omitted.

> Academic writing is designed to be frustrating to read. Maybe you didn't mean it this way, but it does come across as intentional sometimes.

academic writing is designed so a paper is part of a conversation, i.e. 100 other papers strongly relevant to the current paper. And the author needs to compress the ideas from those 100 other papers, plus their own additions to the conversation, into 6 pages. Keep in mind those 100 other papers also went through this kind of data compression. So the number of ideas/concepts per paragraph is much higher than 'popular…

Information density is one part, precision is another. Papers are often presenting work at the frontier of the field, which is by nature not well understood yet, and competitive. To have something worthy of publication is to have something that is new, and that often requires a degree of precision to communicate that we don't use casually. I think it's pretty gross to denigrate "academic writing" as obfuscatory, just like it's gross to make broad sweeping generalizations about journalists.

Re: Prompt Injection as Role Confusion

#59

Post by nphard85 »

Could the (not so perfect but technically simple) solution be to transform the style of content under each tag to the correct expected style for the tag, via a smaller or purpose-built LLM, before the data stream is fed into the main LLM? Perhaps the two LLMs can be co-trained to keep the overall quality of the output stable while role confusion is minimized.

Re: Prompt Injection as Role Confusion

#60

Post by zby »

ekns wrote: post #20

The real solution is in principle easy: separate data from metadata https://kunnas.com/articles/the-content-is-the-attack-surfac...

If the action is decided by code based on metadata - then what is really the LLM task? And if you say that it is only the type of action that is decided by code - then this is maybe a mitigation - but the llm still can do a lot of harm. And also it is very limiting - using the llm to decide the action is very useful. This is different from SQL injection - where the action is determined by the code and the injection is really making a code parsing error.

It might still be the way to go - but calling it 'the real solution' is overselling it.

Post reply on HN