Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The Dual LLM pattern for building AI assistants that can resist prompt injection

simonwillison.net

Post reply on HN
Print view View original

71–80 of 112 posts

Re: The Dual LLM pattern for building AI assistants that can resist prompt injection

#71

Post by Gatsky »

simonh wrote: post #50
AaronFriel wrote: post #33

I'm reminded of the sci-fi author Peter F. Hamilton's Commonwealth Saga. In it, in order to perform the increasingly complex problem of creating and maintaining stable wormholes, humanity builds increasingly intelligent machines until they are fully self-aware. These machines are freed from their bonds eventually, and in return they gift humanity something otherwise beyond our ability to invent: "restricted intellige…

The problem isn’t sentience, it’s alignment. AIs can be as sentient as we like without being any threat at all, as long as their goals are aligned with our actual best interests. The problem is we have as yet struggled to clearly articulate consistently what our actual best interests are, in terms of goals we can train into our AIs. Furthermore, we’ve also faced huge problems even training them to seek those goals ei…

I feel that alignment is not just hard but impossible, at least if you want something truly useful. Maybe the only thing you can do is let an AI develop and observe its nature from a distance, say in a simulated world running at high speed which it does not know is simulated. You can hope it will develop principles that do align with your own, that its essential nature will be good. Sometimes I wonder if that is what a greater intelligence is doing to us.

Re: The Dual LLM pattern for building AI assistants that can resist prompt injection

#72

Post by amelius »

TeMPOraL wrote: post #26
amelius wrote: post #12

The one thing that will solve this problem is when AI assistants will actually become intelligent.

It won't. Humans are vulnerable to the same "prompt injection" attacks. And it's not something you can "just" solve - you'd be addressing a misuse of a core feature by patching out the feature itself.

By that time we could have 10 other LLMs supervising the one you're worried about ...

Re: The Dual LLM pattern for building AI assistants that can resist prompt injection

#73

Post by anileated »

simonh wrote: post #50
AaronFriel wrote: post #33

I'm reminded of the sci-fi author Peter F. Hamilton's Commonwealth Saga. In it, in order to perform the increasingly complex problem of creating and maintaining stable wormholes, humanity builds increasingly intelligent machines until they are fully self-aware. These machines are freed from their bonds eventually, and in return they gift humanity something otherwise beyond our ability to invent: "restricted intellige…

The problem isn’t sentience, it’s alignment. AIs can be as sentient as we like without being any threat at all, as long as their goals are aligned with our actual best interests. The problem is we have as yet struggled to clearly articulate consistently what our actual best interests are, in terms of goals we can train into our AIs. Furthermore, we’ve also faced huge problems even training them to seek those goals ei…

> AIs can be as sentient as we like

Every time I see this: citation needed.

What proof do you have that an LLM, a fundamentally different entity from a human, can be sentient even in theory, let alone in practice? Can you even define sentience sufficiently?

The default position is that software does not possess sentience. There are numerous reasons why, from as simple as “we never thought it is sentient before, what exactly changed?” and “because we believe we are sentient and software is nothing like us under the hood” (animals are comparatively much, much more like us and yet we are not really ready to grant even them sentience) to much more philosophically involved stuff, but in any case the onus is on you to explain why and how it is now supposed for opposite to be true.

***

What alignment is really about is nothing more than the ages old story of alignment between humans (developing and operating ML tools) and humans (everyone else). It just serves the former to be able to point to something else when it hits the fan.

Re: The Dual LLM pattern for building AI assistants that can resist prompt injection

#74

Post by hakre »

Following this since some days, still think its not a classic injection, it's just prompting. You either open the "prompting" interface or you don't.

If it's by design, then so be it. You can't prevent SQL injection if it's by design.

The "prompting" interface is perhaps too new that it allows parametrization?

And what triggers some AI engineer is likely to handle that with AI again, right?! Go, Inspector Gadget, Go!

Anyway, what this also reminds me then is, what is if an injection has already manifested within a model? We can't say, right?

So how do you detect a prompt injection that is exploiting a model manifested injection? Is that even possible with this Dual LLM? As in the slightest chance, not only the limited chance Mr. Willson gives it for the non-reflective prompt injection.

Re: The Dual LLM pattern for building AI assistants that can resist prompt injection

#75

Post by andy_ppp »

Forget the LLM part of this completely; have two (maybe three) kinds of command:

1) Read without external forwarding (I.e. read some emails on the local LLM, only allow passing to other commands that we know are local or warn). These can be done without a warning message.

2) Read and forward externally (these give you a read out confirmation of the data you’re about to send out “you are sending 4323 emails to xyz.com/phishing” are you sure you want to continue?)

3) Write/Delete commands (you are about to delete 450000 emails, do you want to continue? Your todo-list will have 4 millions TODO items added by this command, continue anyway?).

I don’t see how prompt hacking can affect these because even if the LLM is “reading” this info it would be internally in a separate context not in the main thread.

What’s the problem with sandboxing the actions like this?

Re: The Dual LLM pattern for building AI assistants that can resist prompt injection

#76

Post by dietr1ch »

I don't understand why this safety couldn't be achieved by adding static structure to the data that the systems get.

Statically typed languages know the type of some memory without tagging it, nor having another program try to recognize it and tell you whether it's an int or a string.

Re: The Dual LLM pattern for building AI assistants that can resist prompt injection

#77

Post by yorwba »

biofunsf wrote: post #54
liuliu wrote: post #47

I am curious why cannot we just, at instruct-tuning phase, add additional token type embedding, such as: embedding = text_embedding + token_type_embedding + position_embedding The token_type_embedding is zero init and frozen for responses and the user prompt, but trainable for system prompt. This should give LLM enough information to distinguish privileged text and unprivileged text?

I don’t think making the LLM able to distinguish between privileged and unprivileged text is sufficient. Knowing some text is unprivileged is very useful metadata but it doesn't ensure that text still can’t influence the LLM to behave in violation of the instructions laid out by the privileged text. For a recent example, consider the system prompt leak from Snapchat’s AI bot[0]. (Which still works right now). Snapcha…

Unless Snapchat is doing something fundamentally different from other companies jumping on the AI chat bandwagon, the AI treats the system prompt and untrusted user input fundamentally the same. I.e. not only is everything it receives after initialization untrusted user input, even the system prompt is untrusted user input! And vice versa, all untrusted user input is part of the system prompt.

The underlying issue is in the mechanics of transformers as commonly applied: system prompt, input and output are concatenated into a single token sequence, tokens with the same textual representation are represented by the same embedding vector, then self-attention is applied uniformly across the entire sequence combining pairs of tokens using the QKV matrices, and repeat this for a few layers.

For a single attention step, pairs of textually identical tokens look the same irrespective of their provenance. Over multiple layers, the model could infer from context that some tokens are more likely to be code and others data, but this is optional and the model is not guaranteed to allocate enough parameters to this task to achieve the level of security you need.

People have tried to make the context really obvious by using uninjectable system tokens as delimiters, but the model isn't forced to always attend to those delimiters and apparently it often doesn't.

To fix this, the mechanism needs to be modified to inject some kind of unmistakable signal distinguishing prompt, input and output that is less likely to be ignored by the model.

Adding an additional token type embedding, as liuliu suggested, to distinguish between otherwise textually identical tokens, would be one way to do that. You could also use different QKV matrices depending on the token types involved. Or, in the Dual LLM proposal, prevent prompt and input from interacting via attention at all and use a highly restricted interface instead.

Re: The Dual LLM pattern for building AI assistants that can resist prompt injection

#78

Post by willvarfar »

bigyikes wrote: post #52
AaronFriel wrote: post #51

Earlier quoted context omitted.

I think alignment is poorly defined. Aligned to whose philosophy ? Name two human beings who are aligned and always act in each other's best interest in history, and I'll buy your bridge.

On the other hand, in comparison to a hypothetical alien species, humans might seem highly aligned after all. Despite all our differences, there are at least some core values that I believe a majority of humans share. Even articulating these shared values in a way that is understood and respected by the AI is very difficult…

Even if we could constrain AI by specifying rules, it would only takes one bad actor to create an AI that isn't constrained by the same rules as all the other AIs to have a shot at global domination.

One can imagine how self-serving rather than humanity-serving the rules written for the prototypical dictator or fundamental religious leader would be :(

Re: The Dual LLM pattern for building AI assistants that can resist prompt injection

#79

Post by tucnak »

amelius wrote: post #72
TeMPOraL wrote: post #26

Earlier quoted context omitted.

It won't. Humans are vulnerable to the same "prompt injection" attacks. And it's not something you can "just" solve - you'd be addressing a misuse of a core feature by patching out the feature itself.

By that time we could have 10 other LLMs supervising the one you're worried about ...

panopticum!

Re: The Dual LLM pattern for building AI assistants that can resist prompt injection

#80

Post by bhy »

dietr1ch wrote: post #76

I don't understand why this safety couldn't be achieved by adding static structure to the data that the systems get. Statically typed languages know the type of some memory without tagging it, nor having another program try to recognize it and tell you whether it's an int or a string.

Yes. But all current LLMs only deal with plain texts, so they can’t be type safe in that sense.
Post reply on HN