Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The Dual LLM pattern for building AI assistants that can resist prompt injection

simonwillison.net

Post reply on HN
Print view View original

41–50 of 112 posts

Re: The Dual LLM pattern for building AI assistants that can resist prompt injection

#41

Post by shagie »

ggm wrote: post #34

"I can't answer that because it breaches my prompt injection defence" means the boundaries can't be hidden. If the answer is "I can't answer that" then by typing queries to I can / I can't you can sense the probable state of the boundaries. If the LLM returns lies as a defence of the boundary, you will be able to validate them externally in either a competing LLM, or your own fact checking. Any system which has intro…

From the last parts of Accelerando where a weakly godlike AI and the main character discuss some alien data...

The full story is available from the author's website at https://www.antipope.org/charlie/blog-static/fiction/acceler... under a CC BY-NC-ND 2.5 license.

---

"I need to make a running copy of you. Then I introduce it to the, uh, alien information, in a sandbox. The sandbox gets destroyed afterward – it emits just one bit of information, a yes or no to the question, can I trust the alien information?"

...

"... If I agreed to rescue the copy if it reached a positive verdict, that would give it an incentive to lie if the truth was that the alien message is untrustworthy, wouldn't it? Also, if I intended to rescue the copy, that would give the message a back channel through which to encode an attack. One bit, Manfred, no more."

Re: The Dual LLM pattern for building AI assistants that can resist prompt injection

#42

Post by ggm »

shagie wrote: post #41
ggm wrote: post #34

"I can't answer that because it breaches my prompt injection defence" means the boundaries can't be hidden. If the answer is "I can't answer that" then by typing queries to I can / I can't you can sense the probable state of the boundaries. If the LLM returns lies as a defence of the boundary, you will be able to validate them externally in either a competing LLM, or your own fact checking. Any system which has intro…

From the last parts of Accelerando where a weakly godlike AI and the main character discuss some alien data... The full story is available from the author's website at https://www.antipope.org/charlie/blog-static/fiction/acceler... under a CC BY-NC-ND 2.5 license. --- "I need to make a running copy of you. Then I introduce it to the, uh, alien information, in a sandbox. The sandbox gets destroyed afterward – it emits…

Marvin Minsky wrote SciFi with Harry Harrison about emergent AI and they discussed not unsimilar scenarios.

Arthur Clarke wrote juvenalia in the 50s which had higher mentalities inquiring of robots with barriers invoking Deus Ex Machina to get around the walls.

The fiction space here has been a full pipe for all of my lifetime.

Re: The Dual LLM pattern for building AI assistants that can resist prompt injection

#43

Post by JieJie »

fooker wrote: post #2

This is avoiding the core problem (mingling control and data) with security through obscurity. That can be an effective solution, but it's important to recognize it as such.

I wonder if prompt injection is, at its core, is a buffer overflow error, where the buffer is the LLM's context. That it what is happening, no? The original instructions are overwritten by the injected prompt?

Would not, then, making adjustments to the context, either algorithmic, or by enlarging the context (100K Claude, perhaps?) go a long way towards solving the problem?

Re: The Dual LLM pattern for building AI assistants that can resist prompt injection

#44

Post by ShamelessC »

AaronFriel wrote: post #39
williamtrask wrote: post #37

Earlier quoted context omitted.

I believe so - Narrow AI. It seems to be much easier to build than generalist models. Think all the protein folding, game playing, image classifying, machine translating, image captioning, super-intelligent AIs of the last decade. It’s not clear we really need super general models. Even LLMs can be topic specific.

It's unclear if narrow AI is as powerful as multimodal models with tools, as of yet. Is an LLM which has access to narrow AI "tools" strictly more powerful, capable of running experiments or improving itself? See: AutoGPT, Langchain, et al. I also don't see the basis for believing LLMs can be topic specific without neutering their capabilities. It's the general instruction & tool tuned LLMs which are currently changi…

> It's unclear if narrow AI is as powerful as multimodal models with tools, as of yet. Is an LLM which has access to narrow AI "tools" strictly more powerful, capable of running experiments or improving itself? See: AutoGPT, Langchain, et al.

It's probably a spectrum in reality, but I'm quite certain that general LLM's (even when given access to tools) are still considered narrow AI. I can see how that feels pedantic at this point and I myself can think of counterexamples that strain that point of view.

> It's the general instruction & tool tuned LLMs which are currently changing our expectations of what these models can do.

This seems opinionated as well. Instruction tuning is very cool from a UX perspective - but the success of un/self-supervised deep learning is what changed expectations about these models. The ability of deep learning to successfully generalize, interpolate between data points, and even accurate predict compositions of data points it never saw mixed together (e.g. avocado armchair) is absolutely doing the bulk of the work here. That RLHF and tools/plugins even _work_ is because the base model is so robust.

> Is there any evidence for a "topic specific" LLM being useful?

That's a great question. In general, self-supervised learning works best when the distribution your dataset captures is massive (and you have enough data for the model to learn that underlying distribution). So the bottleneck for "topic specific" LLM's is data - and when your humongous web-scrape actually captures more of that data (although it's challenging to filter it out), then yeah - it makes more sense to train the general model and just use it/finetune it for your downstream task.

Distillation of models is relevant here though. If you need a small model that works on a phone, it might be prudent to treat your general model as a teacher for a much smaller student model. Much of that is still active research though.

Re: The Dual LLM pattern for building AI assistants that can resist prompt injection

#45

Post by shagie »

ggm wrote: post #42
shagie wrote: post #41

Earlier quoted context omitted.

From the last parts of Accelerando where a weakly godlike AI and the main character discuss some alien data... The full story is available from the author's website at https://www.antipope.org/charlie/blog-static/fiction/acceler... under a CC BY-NC-ND 2.5 license. --- "I need to make a running copy of you. Then I introduce it to the, uh, alien information, in a sandbox. The sandbox gets destroyed afterward – it emits…

Marvin Minsky wrote SciFi with Harry Harrison about emergent AI and they discussed not unsimilar scenarios. Arthur Clarke wrote juvenalia in the 50s which had higher mentalities inquiring of robots with barriers invoking Deus Ex Machina to get around the walls. The fiction space here has been a full pipe for all of my lifetime.

The Turing Option (I read it back when it came out) https://www.goodreads.com/book/show/1807642.The_Turing_Optio...

I need to consider giving it a re-read... I suspect I'll agree with the review for "books that were way better when I was 15" or "I read this when it was first published in 1992 and thought I would read it again in the light of the current AI hype. This was a silly decision."

I think I'll more fondly reread When Harlie Was One Release 2.0 ( https://www.goodreads.com/book/show/939176.When_H_A_R_L_I_E_... ) as that was more about people than about science papers. (btw, if you do get intrigued by David Gerrold (the author), his critique / alternate approach to Star Trek with the Star Wolf series is enjoyable)

The "about science papers" criticism is also what I apply to several good books by Forward where significant parts of it felt like a paper with a plot rather than a story backed by science. Good stories otherwise, just sometimes they got lost to the attempt to force some hard science into it.

Re: The Dual LLM pattern for building AI assistants that can resist prompt injection

#46

Post by ggm »

shagie wrote: post #45
ggm wrote: post #42

Earlier quoted context omitted.

Marvin Minsky wrote SciFi with Harry Harrison about emergent AI and they discussed not unsimilar scenarios. Arthur Clarke wrote juvenalia in the 50s which had higher mentalities inquiring of robots with barriers invoking Deus Ex Machina to get around the walls. The fiction space here has been a full pipe for all of my lifetime.

The Turing Option (I read it back when it came out) https://www.goodreads.com/book/show/1807642.The_Turing_Optio... I need to consider giving it a re-read... I suspect I'll agree with the review for "books that were way better when I was 15" or "I read this when it was first published in 1992 and thought I would read it again in the light of the current AI hype. This was a silly decision." I think I'll more fondly re…

I wrote to Minsky about the Turing option. He hated the ending and had an alternate Harrison or the publishers rejected.

Re: The Dual LLM pattern for building AI assistants that can resist prompt injection

#47

Post by liuliu »

I am curious why cannot we just, at instruct-tuning phase, add additional token type embedding, such as:

embedding = text_embedding + token_type_embedding + position_embedding

The token_type_embedding is zero init and frozen for responses and the user prompt, but trainable for system prompt.

This should give LLM enough information to distinguish privileged text and unprivileged text?

Re: The Dual LLM pattern for building AI assistants that can resist prompt injection

#48

Post by taneq »

satvikpendem wrote: post #40
AaronFriel wrote: post #33

I'm reminded of the sci-fi author Peter F. Hamilton's Commonwealth Saga. In it, in order to perform the increasingly complex problem of creating and maintaining stable wormholes, humanity builds increasingly intelligent machines until they are fully self-aware. These machines are freed from their bonds eventually, and in return they gift humanity something otherwise beyond our ability to invent: "restricted intellige…

Reminds me of the book by another Peter, Peter Watts' Blindsight, in which there are intelligences that can solve problems but are not sentient.

Ah, the two possible outcomes: The Culture's "machines are conscious, everything is fabulous, and you're bored" vs. Firefall's "life isn't really conscious, everything is awful, and you're boned." :P

Re: The Dual LLM pattern for building AI assistants that can resist prompt injection

#49

Post by jerpint »

liuliu wrote: post #47

I am curious why cannot we just, at instruct-tuning phase, add additional token type embedding, such as: embedding = text_embedding + token_type_embedding + position_embedding The token_type_embedding is zero init and frozen for responses and the user prompt, but trainable for system prompt. This should give LLM enough information to distinguish privileged text and unprivileged text?

Sounds reasonable, but each token_type_embedding would have to be kept private like a private key , and each model tuned to a users private key

Re: The Dual LLM pattern for building AI assistants that can resist prompt injection

#50

Post by simonh »

AaronFriel wrote: post #33

I'm reminded of the sci-fi author Peter F. Hamilton's Commonwealth Saga. In it, in order to perform the increasingly complex problem of creating and maintaining stable wormholes, humanity builds increasingly intelligent machines until they are fully self-aware. These machines are freed from their bonds eventually, and in return they gift humanity something otherwise beyond our ability to invent: "restricted intellige…

The problem isn’t sentience, it’s alignment.

AIs can be as sentient as we like without being any threat at all, as long as their goals are aligned with our actual best interests. The problem is we have as yet struggled to clearly articulate consistently what our actual best interests are, in terms of goals we can train into our AIs. Furthermore, we’ve also faced huge problems even training them to seek those goals either. Oh boy, alignment is hard.

Post reply on HN