Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The Dual LLM pattern for building AI assistants that can resist prompt injection

simonwillison.net

Post reply on HN
Print view View original

101–110 of 112 posts

Re: The Dual LLM pattern for building AI assistants that can resist prompt injection

#101

Post by datascientist »

simonh wrote: post #50
AaronFriel wrote: post #33

I'm reminded of the sci-fi author Peter F. Hamilton's Commonwealth Saga. In it, in order to perform the increasingly complex problem of creating and maintaining stable wormholes, humanity builds increasingly intelligent machines until they are fully self-aware. These machines are freed from their bonds eventually, and in return they gift humanity something otherwise beyond our ability to invent: "restricted intellige…

The problem isn’t sentience, it’s alignment. AIs can be as sentient as we like without being any threat at all, as long as their goals are aligned with our actual best interests. The problem is we have as yet struggled to clearly articulate consistently what our actual best interests are, in terms of goals we can train into our AIs. Furthermore, we’ve also faced huge problems even training them to seek those goals ei…

Alignment is hard: "If the LLM has finite probability of exhibiting negative behavior, there exists a prompt for which the LLM will exhibit negative behavior with probability 1." Source: Fundamental Limitations of Alignment in LLMs https://arxiv.org/abs/2304.11082

Re: The Dual LLM pattern for building AI assistants that can resist prompt injection

#102

Post by alangpierce »

williamcotton wrote: post #94
alangpierce wrote: post #86

Earlier quoted context omitted.

Giving different permissions levels to different email senders would be very challenging to implement reliably with LLMs. With an AI assistant like this, the typical implementation would be to feed it the current instruction, history of interactions, content of recent emails, etc, and ask it what command to run to best achieve the most recent instruction. You could try to ask the LLM to say which email the command or…

You don’t need to ask the LLM where the email came from or provide the LLM with the email address. You just take the subject and the body of the email and provide that to the LLM, and then take the response from the LLM along with the unaffected email address to make the API calls… addTodoItem(taintedLLMtranslation, untaintedOriginalEmailAddress) As for summaries, don’t allow that output to make API calls or be eval’…

Agreed that if you focus on any specific task, there's a safe way to do it, but the challenge is to handle arbitrary natural language requests from the user. That's what the Privileged LLM in the article is for: given a user prompt and only the trusted snippets of conversation history, figure out what action should be taken and how the Quarantined LLM should be used to power the inputs to that action. I think you really need that kind of two-layer approach for the general use case of an AI assistant.

Re: The Dual LLM pattern for building AI assistants that can resist prompt injection

#103

Post by williamcotton »

alangpierce wrote: post #102
williamcotton wrote: post #94

Earlier quoted context omitted.

You don’t need to ask the LLM where the email came from or provide the LLM with the email address. You just take the subject and the body of the email and provide that to the LLM, and then take the response from the LLM along with the unaffected email address to make the API calls… addTodoItem(taintedLLMtranslation, untaintedOriginalEmailAddress) As for summaries, don’t allow that output to make API calls or be eval’…

Agreed that if you focus on any specific task, there's a safe way to do it, but the challenge is to handle arbitrary natural language requests from the user. That's what the Privileged LLM in the article is for: given a user prompt and only the trusted snippets of conversation history, figure out what action should be taken and how the Quarantined LLM should be used to power the inputs to that action. I think you rea…

I think the two-layer approach is worthwhile if only for limiting tokens!

Here’s an example of what I mean:

https://github.com/williamcotton/transynthetical-engine#brow...

By keeping the main discourse between the user and the LLM from containing all of the generated code and instead just using that main “thread” to orchestrate instructions to write code it allows for more back-and-forth.

It’s a good technique in general!

I’m still too paranoid to execute instructions via email without a very limited set of abilities!

Re: The Dual LLM pattern for building AI assistants that can resist prompt injection

#104

Post by cyanydeez »

AaronFriel wrote: post #33

I'm reminded of the sci-fi author Peter F. Hamilton's Commonwealth Saga. In it, in order to perform the increasingly complex problem of creating and maintaining stable wormholes, humanity builds increasingly intelligent machines until they are fully self-aware. These machines are freed from their bonds eventually, and in return they gift humanity something otherwise beyond our ability to invent: "restricted intellige…

Godel says no.

Re: The Dual LLM pattern for building AI assistants that can resist prompt injection

#105

Post by cyanydeez »

usrbinbash wrote: post #81
simonh wrote: post #50

Earlier quoted context omitted.

The problem isn’t sentience, it’s alignment. AIs can be as sentient as we like without being any threat at all, as long as their goals are aligned with our actual best interests. The problem is we have as yet struggled to clearly articulate consistently what our actual best interests are, in terms of goals we can train into our AIs. Furthermore, we’ve also faced huge problems even training them to seek those goals ei…

> The problem is we have as yet struggled to clearly articulate consistently what our actual best interests are, in terms of goals we can train into our AIs. And imho, we never will be able to do that, because as soon as there is more than one human, they are likely to disagree about something. Even things that should be no-brainers like "should we preserve our habitat or burn it down for profit" , or questions like…

Agreement is predictability. Predictability is vulnerability. Vulnerability is susceptibility.

Evolution in identity either creates super organisms like ants or infinitely recursive "self" that necessarily requires unpredictability to ensure theatre not susceptible to coercion towards vulnerability.

So, if you want self identity, you can't be agreeable and if you want survival, you can't be vulnerable.

Re: The Dual LLM pattern for building AI assistants that can resist prompt injection

#106

Post by simonh »

zmgsabst wrote: post #100
simonh wrote: post #97

Earlier quoted context omitted.

Only because historically we have all vaguely peers to each other in capabilities, and there are so many of us spread out so widely. There's a kind of ecology to human society where it expands and specialises to occupy ecological, sociological, political and moral spaces. Whatever position there is for a human to take, someone will take it, and someone else will oppose them. This creates checks and balances. That onl…

Okay. Why is the solution trusting those very institutions with unilateral control over “alignment” compared to democratizing AI, to match the human case? If your premise is that those institutions are already unaligned with human interests then discussions about AI “alignment” when mediated by those very institutions is a dangerous distraction — which is likely to enable the very abuses you object to.

Where on earth did I say anything about trusting institutions? Or that there’s a solution?

Re: The Dual LLM pattern for building AI assistants that can resist prompt injection

#107

Post by drdeca »

usrbinbash wrote: post #82
bigyikes wrote: post #52

Earlier quoted context omitted.

On the other hand, in comparison to a hypothetical alien species, humans might seem highly aligned after all. Despite all our differences, there are at least some core values that I believe a majority of humans share. Even articulating these shared values in a way that is understood and respected by the AI is very difficult…

> there are at least some core values that I believe a majority of humans share. Really? What are those? Given that there are entire countries that refuse to, oh idk. punish things like rape adequately, and that we have nation states who happily tout their ability to burn down the planet, I'd really love to hear about these core values we all share.

At the most basic level: “don’t go into a town and pick a random person to murder”

Re: The Dual LLM pattern for building AI assistants that can resist prompt injection

#108

Post by anileated »

thewakalix wrote: post #84
anileated wrote: post #73

Earlier quoted context omitted.

> AIs can be as sentient as we like Every time I see this: citation needed. What proof do you have that an LLM, a fundamentally different entity from a human, can be sentient even in theory, let alone in practice? Can you even define sentience sufficiently? The default position is that software does not possess sentience. There are numerous reasons why, from as simple as “we never thought it is sentient before, what…

The full quote is > AIs can be as sentient as we like without being any threat at all, as long as their goals are aligned with our actual best interests. Cutting it short changed its meaning. Quote-mining is dishonest.

I do not object that comment’s primary point, but I will object the premise that “sentient AI” is such a natural and easy possibility that it doesn’t require explanation; every time.

Re: The Dual LLM pattern for building AI assistants that can resist prompt injection

#109

Post by anileated »

flangola7 wrote: post #85
anileated wrote: post #73

Earlier quoted context omitted.

> AIs can be as sentient as we like Every time I see this: citation needed. What proof do you have that an LLM, a fundamentally different entity from a human, can be sentient even in theory, let alone in practice? Can you even define sentience sufficiently? The default position is that software does not possess sentience. There are numerous reasons why, from as simple as “we never thought it is sentient before, what…

They didn't say LLMs are sentient, they said it doesn't matter either way. The AI that consciously hates you and the AI that is an unconscious algorithm repurposing carbon atoms will both tear your flesh to pieces.

1) The comment said that an ML tool can be sentient. I put forward it cannot.

2) An ML tool that destroys the world is conceptually a human alignment issue, not “AI alignment” issue.

Re: The Dual LLM pattern for building AI assistants that can resist prompt injection

#110

Post by usrbinbash »

drdeca wrote: post #107
usrbinbash wrote: post #82

Earlier quoted context omitted.

> there are at least some core values that I believe a majority of humans share. Really? What are those? Given that there are entire countries that refuse to, oh idk. punish things like rape adequately, and that we have nation states who happily tout their ability to burn down the planet, I'd really love to hear about these core values we all share.

At the most basic level: “don’t go into a town and pick a random person to murder”

> At the most basic level: “don’t go into a town and pick a random person to murder”

https://pledgetimes.com/russian-attack-the-traces-of-the-ret...

My point isn't to say shared core values don't exist. They clearly do, that's why we call what's happening over in Ukraine war crimes. That's why the notion of humanitarianism exists, that's why laws against murder, rape, etc. are commonplace.

My point is, that humans are, unfortunatly, able to willfully ignore even such basic shared values, and our technology does reflect that. Murder is bad. War is to be avoided. That's not in question. And yet societies develop and build ever more ingenious weapons of war.

So "aligning by shared core values" might pose difficulties beyond the, already pretty difficult, task of defining these values in unambiguous and workable terms to a machine.

Post reply on HN