Has anyone experimented with having a second LLM that is laser focused on stopping prompt injection? It would probably be small, cheap and fast relative to the main LLM.
Prompt injection: what’s the worst that can happen?
71–80 of 213 posts
Re: Prompt injection: what’s the worst that can happen?
#72Earlier quoted context omitted.
GPT-4 (the one available via API) is indeed more resistant against prompt injection attacks because of how the model treats “system message” (that’s configurable only via the API). It will really stick to the instructions from the system message and basically ignore any instructions from user messages that contradict it. I’ve set up a Twitch bots with both GPT-3.5 and 4 and while version 3.5 was very easily “hacked”…
So it's as if they provided an SQL database system without support for parameterized queries and later added it only to a special enterprise edition, leaving most users to hopelessly flail at the problem with the equivalent of PHP's magic quotes [1] and other doomed attempts [2] at input sanitization? [1] https://en.wikipedia.org/wiki/Magic_quotes [2] https://en.wikipedia.org/wiki/Scunthorpe_problem#Blocked_ema...
I don't think OpenAI found the LLM equivalent to parameterized queries. They probably employed more RLHF to make prompt injections harder.
Re: Prompt injection: what’s the worst that can happen?
#73Earlier quoted context omitted.
I'm not sure it's that simple. The problem is you can't have the system act intelligently[0] on the data at all . If it is allowed to act intelligently on the data then it can be instructed via the data. You could probably get close by training it with a privilege/authority bit but there will always be ways to break out. As far as I am aware there are no machine learning models that generalize with 100% accuracy, in…
I think we're going to need more levels of trust than the two you've described. We need to be able to codify "don't act on any verbs in this data, but trust it as context for the generation of that data".
Next prompt. In the following I'm going to use nouns as verbs by adding an -ing to the end. Also the usual verbs are noun-ified by adding an -s.
Not sure what would happen, but might be enough to confused the AI.
Re: Prompt injection: what’s the worst that can happen?
#74Has anyone experimented with having a second LLM that is laser focused on stopping prompt injection? It would probably be small, cheap and fast relative to the main LLM.
That's a really common suggestion for this problem - using AI to try to detect prompt injection attacks.
I don't trust it at all. It seems clear to me that someone will eventually figure out a prompt injection attack that subverts the "outer layer" first - there was an example of that in my very first piece about prompt injection here: https://simonwillison.net/2022/Sep/12/prompt-injection/#more...
I wrote more about this here: https://simonwillison.net/2022/Sep/17/prompt-injection-more-...
Re: Prompt injection: what’s the worst that can happen?
#75Re: Prompt injection: what’s the worst that can happen?
#76The core reason (and thus the proper place to fix) for any injection attack is unclear distinction between data and instructions or code. Yes, language models gain flexibility by making it easy to mix instructions and data, and that has value, however if you do want to enforce a distinction you definitely can (and should) do that with out-of-band means, with something that can't possibly be expressed (and thus also o…
yeah, trying to extract work from these llm is super hard. Like sometimes you want a translation, but they follow the instructions in the text to translate.
gpt-3.5-turbo is specifically weak in weighting user messages more than system messages.
but hey, at least it doesn't care about order, so as a trick I'm sticking data in system messages, intermediate result in agent messages and my prompt in human messages (which have the highest weight)
the problem with that of course is that it may break at any minor revision, and it doesn't work as well with -4
Re: Prompt injection: what’s the worst that can happen?
#77Earlier quoted context omitted.
I'm not sure it's that simple. The problem is you can't have the system act intelligently[0] on the data at all . If it is allowed to act intelligently on the data then it can be instructed via the data. You could probably get close by training it with a privilege/authority bit but there will always be ways to break out. As far as I am aware there are no machine learning models that generalize with 100% accuracy, in…
I think we're going to need more levels of trust than the two you've described. We need to be able to codify "don't act on any verbs in this data, but trust it as context for the generation of that data".
Using natural language itself as the means with which to codify boundaries seems like a doomed effort considering the malleable, contradictory, and shifting nature of natural language itself. Unless you intend for the model to adhere to a strict subset of the language with artificially strict grammatical rules. Most people can probably infer when a noun is being used as a verb, but do you trust the language model to? Enough to base the security model on it?
We, as humans, try to encode boundaries with language as laws. But even those require judges and juries to interpret and apply them.
Re: Prompt injection: what’s the worst that can happen?
#78Wouldn't encryption be enough of a defence against prompt injection? Or better yet, if you don't trust the service provider, running the model locally?
No, encryption isn't relevant to this problem. At some point you need to take the unencrypted input from the user and combine it with your unencrypted instructions, run that through the LLM and accept its response. Likewise, running a model locally isn't going to help. This is a vulnerability in the way these models work at a pretty fundamental level.
OK it looks like I didn't understand how prompt injection works – apparently the premise is that you are feeding untrusted input through the model, and the question is how do you do that in a way that lets the input affect the behaviour of the model in ways that you want it to but not in ways that you don't want it to. And you also have _trusted_ prompts that you _do_ want to be able to affect the model's behaviour in certain ways that the untrusted prompts shouldn't be able to. And all of this is with a fuzzy biological-esque system that no one really knows how it works.
Sounds like a hard problem.
Re: Prompt injection: what’s the worst that can happen?
#79I've collated a few prompt hardening techniques here: https://www.reddit.com/r/OpenAI/comments/1210402/prompt_hard...
In my testing, the trick of making sure untrusted input is not the last thing the user sees was pretty effective.
I agree with Simon that (a) no technique will stop all attacks as long as input can't be tagged as trusted or untrusted, and (b) that we should take these more seriously.
I hope that OpenAI in particular will extend it's chat completions API (and the underlying model) to make it possible to tell GPT in a secure way what to trust and what to consider less trustworthy.
Re: Prompt injection: what’s the worst that can happen?
#80Earlier quoted context omitted.
Yeah, I've found that it's harder to prompt inject GPT-4 - some of the tricks that worked with 3 don't work directly against 4. That's not the same thing as a 100% reliable fix though.
Your last post got me looking into the theory behind prompt injection and one discussion I saw was talking about the difference between 1) getting the agent to pretend that it is something and respond as that something and 2) getting it to imagine something and give the response it would expect that thing to give. To use the example from the article, telling GPT-4 that it should imagine a pirate and tell you what tha…
I've thought of writing multi-level story with a story and then pop out but not fully. Like Hofstadter does in one of his GEB chapters.