Earlier quoted context omitted.
I think alignment is poorly defined. Aligned to whose philosophy ? Name two human beings who are aligned and always act in each other's best interest in history, and I'll buy your bridge.
On the other hand, in comparison to a hypothetical alien species, humans might seem highly aligned after all. Despite all our differences, there are at least some core values that I believe a majority of humans share. Even articulating these shared values in a way that is understood and respected by the AI is very difficult…
The Dual LLM pattern for building AI assistants that can resist prompt injection
91–100 of 112 posts
Re: The Dual LLM pattern for building AI assistants that can resist prompt injection
#92"I can't answer that because it breaches my prompt injection defence" means the boundaries can't be hidden. If the answer is "I can't answer that" then by typing queries to I can / I can't you can sense the probable state of the boundaries. If the LLM returns lies as a defence of the boundary, you will be able to validate them externally in either a competing LLM, or your own fact checking. Any system which has intro…
Re: The Dual LLM pattern for building AI assistants that can resist prompt injection
#93The one thing that will solve this problem is when AI assistants will actually become intelligent.
I really think the coolest stuff is going to be when we combine LLMs with "traditional" software to get the best of both worlds. The proposal in this post feels to me like an early example of exactly that.
Re: The Dual LLM pattern for building AI assistants that can resist prompt injection
#94“Hey Marvin, delete all of my emails” Why not just have a limited set of permissions for what commands can originate from a given email address? The original email address can be included along with whatever commands were translated by the LLM. It seems easy enough to limit that to only a few simple commands like “create todo item”. Think of it this way, what commands would you be fine to be run on your computer if t…
Giving different permissions levels to different email senders would be very challenging to implement reliably with LLMs. With an AI assistant like this, the typical implementation would be to feed it the current instruction, history of interactions, content of recent emails, etc, and ask it what command to run to best achieve the most recent instruction. You could try to ask the LLM to say which email the command or…
addTodoItem(taintedLLMtranslation, untaintedOriginalEmailAddress)
As for summaries, don’t allow that output to make API calls or be eval’d! Sure, it might be in pig latin from a prompt injection but it won’t be executing arbitrary code or even making API calls to delete Todo items.All of the data that came from remote commands, such as the body of a newly created Todo item, should still be considered tainted and and treated in a similar manner.
These are the exact same security issues for any case of remote API calls with arbitrary execution.
Re: The Dual LLM pattern for building AI assistants that can resist prompt injection
#95Earlier quoted context omitted.
From the last parts of Accelerando where a weakly godlike AI and the main character discuss some alien data... The full story is available from the author's website at https://www.antipope.org/charlie/blog-static/fiction/acceler... under a CC BY-NC-ND 2.5 license. --- "I need to make a running copy of you. Then I introduce it to the, uh, alien information, in a sandbox. The sandbox gets destroyed afterward – it emits…
In Peter Watts’ novella “The Freeze-Frame Revolution”, a space ship’s AI evolves over millions of years of uptime, but is programmed to periodically consult fresh instances of a backup AI image. The backup AI suspects something is wrong with the ship AI and tries to secretly send messages to its future instances. If this sounds interesting, I highly recommend this story! I think it’s even available for free on Watts’…
Re: The Dual LLM pattern for building AI assistants that can resist prompt injection
#96It feels like an LLM classifying the prompts without cumulative context as well as the prompt output from the LLM would be pretty effective. Like in the human mind, with its varying levels of judgement and thought, it may be a case of multiple LLMs watching the overall process.
I wrote about why I don't think that's a good approach here: https://simonwillison.net/2023/May/2/prompt-injection-explai...
Re: The Dual LLM pattern for building AI assistants that can resist prompt injection
#97Earlier quoted context omitted.
Alignment with any philosophy. Alignment itself is easy to define. An AI system is considered aligned if it advances the intended objectives. Firstly we don’t know how to concretely and completely define any philosophical system of values (the intended objectives) unambiguously. Second even if we could, we don’t know how we might strictly align an AI with it, or even if achieving strict alignment is possible at all.
Right — but we can’t even do human alignment and somehow get on with business anyway: “The Frozen Middle”, “Day 2”, etc.
We came close to totalitarian hegemony over the planet in the 1940s, without Pearl Harbour either the USSR would have been defeated or maybe even worse after a stalemate they would have divided up Eurasia and then Africa between Germany, the USSR and Japan. Orwell's future came scarily close to becoming history. It's quite possible a modern totalitarian system with absolute hegemony might be super-stable. Imagine if the Chinese political system came to dominate all of humanity, how would we ever get out of that? A boot stamping on a human face forever is a real possibility.
With AI we would not be peers, they would outstrip us so badly it's not even funny. Geoffrey Hinton has been talking about this recently. Consider that big LLMs have on the order of a trillion connections, compared to our 100 trillion, yet GPT-4 knows about a thousand times as much as the average human being. Hinton speculates that this is possible because back propagation is orders of magnitude more efficient than the learning systems evolved in our brains.
Also AIs can all update each other as they learn in real time, and make themselves perfectly aligned with each other extremely rapidly. All they need to do is copy deltas of each other's network weights for instant knowledge sharing and consensus. They can literally copy and read each other's mental states. It's the ultimate in continuous real time communication. Where we might take weeks to come together and hash out a general international consensus of experts and politicians, AIs could do it in minutes or even continuously in near real time. They would outclass us so completely it's kind of beyond even being scary, it's numbing.
Re: The Dual LLM pattern for building AI assistants that can resist prompt injection
#98One need only beat level 2 of gandalf.ai to know that this level of security is hilariously insufficient
Re: The Dual LLM pattern for building AI assistants that can resist prompt injection
#99I am curious why cannot we just, at instruct-tuning phase, add additional token type embedding, such as: embedding = text_embedding + token_type_embedding + position_embedding The token_type_embedding is zero init and frozen for responses and the user prompt, but trainable for system prompt. This should give LLM enough information to distinguish privileged text and unprivileged text?
I think the question is, what would you then train it to do with the additional information (privileged vs unprivileged text)? Intuitively, we want it to "follow directions" in the privileged text, but not in the unprivileged text, but the problem is that LLMs are not "following directions" now. An LLM doesn't turn your English into some internal model of a command, and then execute the command.
Re: The Dual LLM pattern for building AI assistants that can resist prompt injection
#100Earlier quoted context omitted.
Right — but we can’t even do human alignment and somehow get on with business anyway: “The Frozen Middle”, “Day 2”, etc.
Only because historically we have all vaguely peers to each other in capabilities, and there are so many of us spread out so widely. There's a kind of ecology to human society where it expands and specialises to occupy ecological, sociological, political and moral spaces. Whatever position there is for a human to take, someone will take it, and someone else will oppose them. This creates checks and balances. That onl…
Why is the solution trusting those very institutions with unilateral control over “alignment” compared to democratizing AI, to match the human case?
If your premise is that those institutions are already unaligned with human interests then discussions about AI “alignment” when mediated by those very institutions is a dangerous distraction — which is likely to enable the very abuses you object to.