Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The Dual LLM pattern for building AI assistants that can resist prompt injection

simonwillison.net

Post reply on HN
Print view View original

51–60 of 112 posts

Re: The Dual LLM pattern for building AI assistants that can resist prompt injection

#51

Post by AaronFriel »

simonh wrote: post #50
AaronFriel wrote: post #33

I'm reminded of the sci-fi author Peter F. Hamilton's Commonwealth Saga. In it, in order to perform the increasingly complex problem of creating and maintaining stable wormholes, humanity builds increasingly intelligent machines until they are fully self-aware. These machines are freed from their bonds eventually, and in return they gift humanity something otherwise beyond our ability to invent: "restricted intellige…

The problem isn’t sentience, it’s alignment. AIs can be as sentient as we like without being any threat at all, as long as their goals are aligned with our actual best interests. The problem is we have as yet struggled to clearly articulate consistently what our actual best interests are, in terms of goals we can train into our AIs. Furthermore, we’ve also faced huge problems even training them to seek those goals ei…

I think alignment is poorly defined. Aligned to whose philosophy? Name two human beings who are aligned and always act in each other's best interest in history, and I'll buy your bridge.

Re: The Dual LLM pattern for building AI assistants that can resist prompt injection

#52

Post by bigyikes »

AaronFriel wrote: post #51
simonh wrote: post #50

Earlier quoted context omitted.

The problem isn’t sentience, it’s alignment. AIs can be as sentient as we like without being any threat at all, as long as their goals are aligned with our actual best interests. The problem is we have as yet struggled to clearly articulate consistently what our actual best interests are, in terms of goals we can train into our AIs. Furthermore, we’ve also faced huge problems even training them to seek those goals ei…

I think alignment is poorly defined. Aligned to whose philosophy ? Name two human beings who are aligned and always act in each other's best interest in history, and I'll buy your bridge.

On the other hand, in comparison to a hypothetical alien species, humans might seem highly aligned after all.

Despite all our differences, there are at least some core values that I believe a majority of humans share. Even articulating these shared values in a way that is understood and respected by the AI is very difficult…

Re: The Dual LLM pattern for building AI assistants that can resist prompt injection

#53

Post by efitz »

There was another post on Thursday related to this [1].

If the LLMs can communicate, then you can use that fact to prompt one to talk to the other and do kind of an indirect injection attack.

[1] https://news.ycombinator.com/item?id=35905876

Re: The Dual LLM pattern for building AI assistants that can resist prompt injection

#54

Post by biofunsf »

liuliu wrote: post #47

I am curious why cannot we just, at instruct-tuning phase, add additional token type embedding, such as: embedding = text_embedding + token_type_embedding + position_embedding The token_type_embedding is zero init and frozen for responses and the user prompt, but trainable for system prompt. This should give LLM enough information to distinguish privileged text and unprivileged text?

I don’t think making the LLM able to distinguish between privileged and unprivileged text is sufficient. Knowing some text is unprivileged is very useful metadata but it doesn't ensure that text still can’t influence the LLM to behave in violation of the instructions laid out by the privileged text.

For a recent example, consider the system prompt leak from Snapchat’s AI bot[0]. (Which still works right now). Snapchat’s AI clearly knows all the subsequent message it receives after initialization are untrusted user input, since for its use case all input is user input. Its system prompt tells it to never reveal the contents of its system prompt. But even then, knowing it’s receiving untrusted input, it still leaks the system prompt.

[0] https://imgur.io/YTOkJ0Y

Re: The Dual LLM pattern for building AI assistants that can resist prompt injection

#55

Post by simonh »

AaronFriel wrote: post #51
simonh wrote: post #50

Earlier quoted context omitted.

The problem isn’t sentience, it’s alignment. AIs can be as sentient as we like without being any threat at all, as long as their goals are aligned with our actual best interests. The problem is we have as yet struggled to clearly articulate consistently what our actual best interests are, in terms of goals we can train into our AIs. Furthermore, we’ve also faced huge problems even training them to seek those goals ei…

I think alignment is poorly defined. Aligned to whose philosophy ? Name two human beings who are aligned and always act in each other's best interest in history, and I'll buy your bridge.

Alignment with any philosophy. Alignment itself is easy to define. An AI system is considered aligned if it advances the intended objectives.

Firstly we don’t know how to concretely and completely define any philosophical system of values (the intended objectives) unambiguously. Second even if we could, we don’t know how we might strictly align an AI with it, or even if achieving strict alignment is possible at all.

Re: The Dual LLM pattern for building AI assistants that can resist prompt injection

#56

Post by simonw »

SeriousGamesKit wrote: post #9

Thanks SimonW! I've really enjoyed your series on this problem on HN and on your blog. I've seen suggestions elsewhere about tokenising fixed prompt instructions differently to user input to distinguish them internally, and wanted to ask for your take on this concept- do you think this is likely to improve the state of play regarding prompt injection, applied either to a one-LLM or two-LLM setup?

I'll believe that works when someone demonstrates it working - it sound good in theory but my hunch is that it's hard or maybe impossible to actually implement.

Re: The Dual LLM pattern for building AI assistants that can resist prompt injection

#57

Post by simonw »

fnordpiglet wrote: post #6

It feels like an LLM classifying the prompts without cumulative context as well as the prompt output from the LLM would be pretty effective. Like in the human mind, with its varying levels of judgement and thought, it may be a case of multiple LLMs watching the overall process.

I wrote about why I don't think that's a good approach here: https://simonwillison.net/2023/May/2/prompt-injection-explai...

Re: The Dual LLM pattern for building AI assistants that can resist prompt injection

#58

Post by charcircuit »

liuliu wrote: post #47

I am curious why cannot we just, at instruct-tuning phase, add additional token type embedding, such as: embedding = text_embedding + token_type_embedding + position_embedding The token_type_embedding is zero init and frozen for responses and the user prompt, but trainable for system prompt. This should give LLM enough information to distinguish privileged text and unprivileged text?

What is unprivileged text?

Re: The Dual LLM pattern for building AI assistants that can resist prompt injection

#59

Post by jamilton »

fooker wrote: post #19
rst wrote: post #13

Earlier quoted context omitted.

It's avoiding the problem by separating control and data, at unknown but signficant cost to functionality (the LLM which determines what tools get invoked doesn't see the actual data or results, only opaque tokens that refer to them, so it can't use them directly to make choices). I'm not sure how that qualifies as "security by obscurity".

It's attempting to split control and data through a system which is susceptible to the same issue. So prompt injection still works, you just have to find the right promt.

Prompt injection with this method could, at worst, make the plaintext incorrect. The summary could be replaced with spam, for example. Prompt injection with the naive method (just have 1 LLM doing everything) could, at worst, directly infect the user's computer.
Post reply on HN