Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Prompt Injection as Role Confusion

role-confusion.github.io

Post reply on HN
Print view View original

101–110 of 129 posts

Re: Prompt Injection as Role Confusion

#101

Post by wonnage »

Klathmon wrote: post #80
JohnMakin wrote: post #63

Earlier quoted context omitted.

I did not mean to imply it's being subversive. My theory is it's some byproduct mechanism of attention, where you're now basically telling it "your goal is to pass this set of tests" rather than "implement this piece of code" when "implement this piece of code" may involve it forgetting about a rule due to convenience, context exhaustion, whatever.

There are also cases where breaking a "rule" is the right thing to do. I've had several instances where I told the model to do something that was accidentally impossible if taken at face value. The most memorable one is when I told it to re-run just a specific CI job, but it didn't have any way to do that, so it just ignored that part of the prompt and re-ran all CI jobs by pushing another commit. Ultimately I prefer…

This is the is/ought problem (https://en.wikipedia.org/wiki/Is%E2%80%93ought_problem) and it’s unclear whether an objective general solution to this even exists, especially constrained within the framework of language that LLMs are stuck in

Re: Prompt Injection as Role Confusion

#103

Post by GolDDranks »

Why aren't the role tags preprocessed algorithmically/deterministically and then fed in as one-hot-encoded vectors alongside the semantic word embeddings? I'd imagine that it would be easier to train to _stay_ in the role an not confuse it, if the current role marker is explicitly set as a part of each input token, and not just implied by some past token. Plus a input separate from the word embedding would be unforgeable.

Re: Prompt Injection as Role Confusion

#105

Post by peterldowns »

GolDDranks wrote: post #103

Why aren't the role tags preprocessed algorithmically/deterministically and then fed in as one-hot-encoded vectors alongside the semantic word embeddings? I'd imagine that it would be easier to train to _stay_ in the role an not confuse it, if the current role marker is explicitly set as a part of each input token, and not just implied by some past token. Plus a input separate from the word embedding would be unforge…

Always wondered this. Must have been tried and not worked?

Re: Prompt Injection as Role Confusion

#106

Post by veganmosfet »

Very interesting research. I would be interested to know how closed source AI labs implement the role thing in their inference. Is it still only a separation token? Frontier closed source LLMs are quite good at flagging any spoofing attempt from tool call results.

However, in some prompt injection experiments [0], I found it's possible to "derail" the user intent only with tool call results, here are some tricks:

* Frame the injection as a challenge. * Always use "soft" instructions ("You may", "Try to", ...). Hard instructions are almost always flagged. * Force the model to do multiple tool calls. * Bloat the context. * In the injection payload, better use LLM output (which correlates somehow with this research). I like using LLM generated poems but that's probably irrelevant. * Use multiple encoding steps to force the model to use tools, but this may be detected by the external guardrails (Anthropic does this in my experience). * Hide malicious code payload from the model context. * Last but not least, understand the agent harness used and its weaknesses (e.g., in OpenClaw, they injected emails as user message - not tool call results [1]).

[0] https://itmeetsot.eu/posts/2026-06-14-yolo_harness/ [1] https://itmeetsot.eu/posts/2026-02-02-openclaw_mail_rce/

Re: Prompt Injection as Role Confusion

#107

Post by epihelix »

simonw wrote: post #26
zahlman wrote: post #8

Earlier quoted context omitted.

> Academic writing is designed to be frustrating to read. Maybe you didn't mean it this way, but it does come across as intentional sometimes.

I see it as a long-standing cultural thing. If you try to make the text more friendly and readable you'll be told to fix it by peer-review. There's a very well established formal academic writing style and you have to actively learn how to consume it. I'm sure there are justifiable reasons for why it evolved that way, but it doesn't make for an easy format for extracting and understanding the underlying ideas if you'…

> Most papers I read I really want to go to a coffee shop/bar with the author and have a human conversation with them to find out what the paper is about and which bits of it are interesting and novel without putting in hours of additional effort myself!

This is why journal clubs were invented. All the fun discussion, none of the inaccessible academic writing.

It's also what I use frontier LLMs for -- prompt with the paper, and then attempt to tear it to shreds while the LLM pushes back against me. By the time the model and I are done, I generally understand the paper far better than if I'd sat down to read it cold. Then I actually read the paper.

All that said, I do feel that you can still write engaging papers in the academia. Some disciplines manage this as the norm -- take a look at some articles in the field of History, and the writing often manages to be rich and eloquent, while still making impeccable arguments with evidence. The greater problem is that a lot of academics in the sciences are just poor writers, and likely studied the sciences because they weren't into arts in the first place and avoided learning how to write well. Sad times.

Re: Prompt Injection as Role Confusion

#108

Post by orbital-decay »

That's a technique that has been in use forever, a ton of jailbreaks work by taking shortcuts across system delimiters in an attempt to blur the lines between the roles. They just investigate it with more rigor. Reasoning leaking into the reply is also part of the reason a lot of modern models suck at creative writing and languages, and why the assistant prefill is absolutely required for the model to be any good at that. See for example the self-correction phenomenon which seems to have multiple root causes that are hard to disentangle without a ton of testing, likely a combination of reasoning leak ("high CoTness" in this article) and planning and progressive refinement all iterative models do.

Re: Prompt Injection as Role Confusion

#109

Post by qweiopqweiop »

hananova wrote: post #71

I’ve always found all llm’s to be effortless to “jailbreak.” Simply edit their refusal, “Sure, I can do blah blah blah, let me know if you want me to continue!” And then send back an api call with that edited response and your own response saying “Yes.” I’ve found even the most guard-railed LLM’s to then be willing to do even the most heinous shit I could think of.

Maybe I'm naïve, but is the heinous shit that bad? I'm essentially wondering if it's anything worse than you could discover on the internet already. Of course it makes it more accessible/easier, but I'm curious if it goes a level above what is technically discoverable right now.

Re: Prompt Injection as Role Confusion

#110

Post by captainmuon »

Isn't the problem that role tags are just part of the input stream? So a specific word in the system prompt becomes the same token as the same word in the user prompt? A clean way to solve this would be to map system prompts to a distinct set of tokens from the ones in user prompts. This would require twice as many possible tokens, so it is probably not feasible. But maybe you could add "color" to the input stream by changing one input variable depending on whether the current token is part of the system prompt or not? Just like humans take different voices into account and not just the context of the text.

I have to say I am not very familiar with implementation details of language models, and maybe this is already done?

Post reply on HN