Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The Dual LLM pattern for building AI assistants that can resist prompt injection

simonwillison.net

Post reply on HN
Print view View original

61–70 of 112 posts

Re: The Dual LLM pattern for building AI assistants that can resist prompt injection

#61

Post by quickthrower2 »

charcircuit wrote: post #58
liuliu wrote: post #47

I am curious why cannot we just, at instruct-tuning phase, add additional token type embedding, such as: embedding = text_embedding + token_type_embedding + position_embedding The token_type_embedding is zero init and frozen for responses and the user prompt, but trainable for system prompt. This should give LLM enough information to distinguish privileged text and unprivileged text?

What is unprivileged text?

Text that, in the examples used to train the neural net, has next token targets that represent answers where the unprivileged text didn’t outsmart the privileged text.

But given that must over or underfit there is no guarantee that it will do perfectly well on test data at honouring this.

Re: The Dual LLM pattern for building AI assistants that can resist prompt injection

#62

Post by quickthrower2 »

The human side to this solution is worrying though. You have an app designed to save you time, and in any such app people will train themselves to “just click it to get it done” almost like a reflex. And so that attack could easily get unnoticed.

You probably need the solution here along with some other heuristics to detect fraud or scams.

e.g. If a friend sent you an email that scores low on how likely it is that they wrote it based on the content then display a red warning and a hidden OK button ala SSL alerts.

For dangerous actions like sending money, delay by 1 hour and send a second factor confirmation that says “you will send money ensure this is not a scam” and only when more questions are answered is it done.

Re: The Dual LLM pattern for building AI assistants that can resist prompt injection

#63

Post by quickthrower2 »

amrb wrote: post #14

So we just recreated all of the previous SQL injection security issues in LLM's, fun times

SQL injection is due to sloppy programming practices and easily avoided. Using something called query parameters.

This is another beast!

Re: The Dual LLM pattern for building AI assistants that can resist prompt injection

#64

Post by quickthrower2 »

williamcotton wrote: post #16

“Hey Marvin, delete all of my emails” Why not just have a limited set of permissions for what commands can originate from a given email address? The original email address can be included along with whatever commands were translated by the LLM. It seems easy enough to limit that to only a few simple commands like “create todo item”. Think of it this way, what commands would you be fine to be run on your computer if t…

Originate from an email address is not secure authentication

Re: The Dual LLM pattern for building AI assistants that can resist prompt injection

#65

Post by cwillu »

JieJie wrote: post #43
fooker wrote: post #2

This is avoiding the core problem (mingling control and data) with security through obscurity. That can be an effective solution, but it's important to recognize it as such.

I wonder if prompt injection is, at its core, is a buffer overflow error, where the buffer is the LLM's context. That it what is happening, no? The original instructions are overwritten by the injected prompt? Would not, then, making adjustments to the context, either algorithmic, or by enlarging the context (100K Claude, perhaps?) go a long way towards solving the problem?

A buffer overflow is a useful reminder that security cuts through abstractions and needs to be built keeping in mind what its fundamentally being built on.

A buffer overflow is fundamentally caused by a separation between allocation and use.

A prompt “injection” is in a real sense a misnomer caused by forgetting what a completion engine does: a prompt can be “injected”, because there is a plausible text that starts with a bunch of text, followed by more text, ultimately ending in (say) the original text repeated. Or transformed. Or whatever. The “emergent common-sense” that is the entire value of a language model is (I suspect) fundamentally in tension with providing restrictions on its output. We can bias the model, but there will always be _some_ weight for _any_ possible output, or else it wouldn't be possible to train the model in the first place.

Re: The Dual LLM pattern for building AI assistants that can resist prompt injection

#66

Post by cwillu »

cwillu wrote: post #65
JieJie wrote: post #43

Earlier quoted context omitted.

I wonder if prompt injection is, at its core, is a buffer overflow error, where the buffer is the LLM's context. That it what is happening, no? The original instructions are overwritten by the injected prompt? Would not, then, making adjustments to the context, either algorithmic, or by enlarging the context (100K Claude, perhaps?) go a long way towards solving the problem?

A buffer overflow is a useful reminder that security cuts through abstractions and needs to be built keeping in mind what its fundamentally being built on. A buffer overflow is fundamentally caused by a separation between allocation and use. A prompt “injection” is in a real sense a misnomer caused by forgetting what a completion engine does: a prompt can be “injected”, because there is a plausible text that starts w…

Which makes me wonder if there's any useful insight from a “0 and 1 are not probabilities” angle: the key being that some classes of output need to somehow be modified to actually have those “probabilities”.

Re: The Dual LLM pattern for building AI assistants that can resist prompt injection

#67

Post by zmgsabst »

simonh wrote: post #55
AaronFriel wrote: post #51

Earlier quoted context omitted.

I think alignment is poorly defined. Aligned to whose philosophy ? Name two human beings who are aligned and always act in each other's best interest in history, and I'll buy your bridge.

Alignment with any philosophy. Alignment itself is easy to define. An AI system is considered aligned if it advances the intended objectives. Firstly we don’t know how to concretely and completely define any philosophical system of values (the intended objectives) unambiguously. Second even if we could, we don’t know how we might strictly align an AI with it, or even if achieving strict alignment is possible at all.

Right — but we can’t even do human alignment and somehow get on with business anyway:

“The Frozen Middle”, “Day 2”, etc.

Re: The Dual LLM pattern for building AI assistants that can resist prompt injection

#68

Post by cpeterso »

shagie wrote: post #41
ggm wrote: post #34

"I can't answer that because it breaches my prompt injection defence" means the boundaries can't be hidden. If the answer is "I can't answer that" then by typing queries to I can / I can't you can sense the probable state of the boundaries. If the LLM returns lies as a defence of the boundary, you will be able to validate them externally in either a competing LLM, or your own fact checking. Any system which has intro…

From the last parts of Accelerando where a weakly godlike AI and the main character discuss some alien data... The full story is available from the author's website at https://www.antipope.org/charlie/blog-static/fiction/acceler... under a CC BY-NC-ND 2.5 license. --- "I need to make a running copy of you. Then I introduce it to the, uh, alien information, in a sandbox. The sandbox gets destroyed afterward – it emits…

In Peter Watts’ novella “The Freeze-Frame Revolution”, a space ship’s AI evolves over millions of years of uptime, but is programmed to periodically consult fresh instances of a backup AI image. The backup AI suspects something is wrong with the ship AI and tries to secretly send messages to its future instances.

If this sounds interesting, I highly recommend this story! I think it’s even available for free on Watts’ website.

Re: The Dual LLM pattern for building AI assistants that can resist prompt injection

#69

Post by cwillu »

bjt2n3904 wrote: post #35

I do believe this is the plot of Portal. Wheatley was created to stop Glados from going on a murderous rampage.

It's also a major plot point of the book “The Golden Transcendence” Book by John C. Wright (part of the series “The Golden Age”)

Re: The Dual LLM pattern for building AI assistants that can resist prompt injection

#70

Post by kyleyeats »

efitz wrote: post #53

There was another post on Thursday related to this [1]. If the LLMs can communicate, then you can use that fact to prompt one to talk to the other and do kind of an indirect injection attack. [1] https://news.ycombinator.com/item?id=35905876

You need a second secret LLM supervisor that's really pulling the strings, rewriting the inputs of the other two.
Post reply on HN