Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Prompt Injection as Role Confusion

role-confusion.github.io

Post reply on HN
Print view View original

11–20 of 129 posts

Re: Prompt Injection as Role Confusion

#11

Post by ryukafalz »

Scene_Cast2 wrote: post #3

Really neat findings. I've personally had a line of thought where you bake in the role into the token. Basically have an embedding (same dim as token dim) for each role, add it to each token. This adds an unambiguous, unspoofable tag. I ran this with a tiny Shakespeare model (not representative) and had a freeform embedding for each speaker. I ended up with a neat similarity map between every character. (I don't thin…

I don't know a ton about how LLMs work (I really should learn), but something like this feels like it might be the way forward to me.

The software running the model knows unambiguously what came from a user and what did not, what came from a tool call and what did not, etc... and having some way of exposing that to the LLM as part of the text itself feels like it fits better with how a neural net works than a set of surrounding tags does.

Re: Prompt Injection as Role Confusion

#12

Post by Scene_Cast2 »

lelanthran wrote: post #7
Scene_Cast2 wrote: post #3

Really neat findings. I've personally had a line of thought where you bake in the role into the token. Basically have an embedding (same dim as token dim) for each role, add it to each token. This adds an unambiguous, unspoofable tag. I ran this with a tiny Shakespeare model (not representative) and had a freeform embedding for each speaker. I ended up with a neat similarity map between every character. (I don't thin…

> I've personally had a line of thought where you bake in the role into the token. Basically have an embedding (same dim as token dim) for each role, add it to each token. This adds an unambiguous, unspoofable tag. Wouldn't this require the training data to also be prepped with the control tokens?

Yes it would. Or, rather, labeling (not extra tokens).

Re: Prompt Injection as Role Confusion

#14

Post by dmazzoni »

Scene_Cast2 wrote: post #3

Really neat findings. I've personally had a line of thought where you bake in the role into the token. Basically have an embedding (same dim as token dim) for each role, add it to each token. This adds an unambiguous, unspoofable tag. I ran this with a tiny Shakespeare model (not representative) and had a freeform embedding for each speaker. I ended up with a neat similarity map between every character. (I don't thin…

My initial thought there is that you'd have an imbalance. Many token patterns would almost never come up with the assistant tag on them, for example words with typos in them.

Re: Prompt Injection as Role Confusion

#16

Post by oli5679 »

Would llms be more robust to this prompt injection if the tags used in fine tuning are sanitised from user input?

E.g. map -> THINK -> USER -> TOOL

If they learn something specific in the chat finetuning stage, this might show LLM its user input text not these tag references.

Re: Prompt Injection as Role Confusion

#18

Post by dvt »

The paper is correct, but I think that anyone that knows anything about LLMs knows this:

> Role tags were a formatting trick that became the security architecture and the cognitive scaffolding of modern LLMs.

LLMs are basically some `f(x) → y` where x and y are strings. That's it. Nothing more to it. If you feed it private x (like secret keys) or do dangerous stuff with y (like running arbitrary non-sandboxed code), that's on you.

Also, roles were never really meant to be a "security architecture," they were just meant to (a) make training/fine-tuning easier, and (b) make conversational LLMs more useful.

Re: Prompt Injection as Role Confusion

#19

Post by tpoacher »

zahlman wrote: post #8
simonw wrote: post #5

> This is a blog-style writeup of the paper YES! I'd love to see more of this. Academic writing is designed to be frustrating to read. Publishing both a paper and a readable blog-style version of it is such a great pattern.

> Academic writing is designed to be frustrating to read. Maybe you didn't mean it this way, but it does come across as intentional sometimes.

I reluctantly confess that I have indeed on occasion had to write in a way that makes the reader have to do a couple of extra mental steps to follow the logic, to avoid reviewers rejecting the manuscript on the grounds of the theoretical contribution being "trivial".

Combine this with added fees for longer papers and you have your answer.

Post reply on HN