Just build GAI and the problem is solved ;)
Prompt injection: what’s the worst that can happen?
121–130 of 213 posts
Re: Prompt injection: what’s the worst that can happen?
#122The core reason (and thus the proper place to fix) for any injection attack is unclear distinction between data and instructions or code. Yes, language models gain flexibility by making it easy to mix instructions and data, and that has value, however if you do want to enforce a distinction you definitely can (and should) do that with out-of-band means, with something that can't possibly be expressed (and thus also o…
It’s better than a free-form input, but that still has to mix input from the prompt and the user in the model. Quality of isolation depends on training, so that’s still a probabilistic result, not a guarantee.
Re: Prompt injection: what’s the worst that can happen?
#123Earlier quoted context omitted.
I'm not sure it's that simple. The problem is you can't have the system act intelligently[0] on the data at all . If it is allowed to act intelligently on the data then it can be instructed via the data. You could probably get close by training it with a privilege/authority bit but there will always be ways to break out. As far as I am aware there are no machine learning models that generalize with 100% accuracy, in…
Yes, the problem here is that what makes pretraining on text so powerful is a double-edged sword: text is, if you will, Turing-complete. People are constantly writing various kinds of instructions, programs, and reasoning or executing algorithms in all sorts of flexible indefinable ways. That's why the models learn so much from text and can do all the things they do, like reason or program or meta-learn or reinforcem…
> text is, if you will, Turing-complete
Text is "Turing Test" -complete :-)
Re: Prompt injection: what’s the worst that can happen?
#124Earlier quoted context omitted.
Yes, the problem here is that what makes pretraining on text so powerful is a double-edged sword: text is, if you will, Turing-complete. People are constantly writing various kinds of instructions, programs, and reasoning or executing algorithms in all sorts of flexible indefinable ways. That's why the models learn so much from text and can do all the things they do, like reason or program or meta-learn or reinforcem…
Yet, even with the current models, with no special tokens, it is relatively straightforward to construct relatively stable defense. Given that the ongoing attack can be detected, tagged and added to a learned filter, this gets a lot less problematic, in practice. Even a simple limit of interactions is effective enough. Here's a practical prompt defense / CTTF that I've made. With five steps of a dialogue limit (per d…
What happens when someone uses software or an LLM to generate a whole fleet of attack prompts?
Re: Prompt injection: what’s the worst that can happen?
#125Could someone explain why you couldn’t just checkpoint the model before responding to the query (assuming the model context changes as a result of the query, otherwise unnecessary), run the query, and then have another model evaluate if the response was in compliance with the original rules. If it was not, roll back to the checkpoint. Otherwise, proceed. You couldn’t prompt inject the regulator as it is not taking user input, so it’s just a matter of instructing the regulator properly, which should be quite feasible.
Re: Prompt injection: what’s the worst that can happen?
#126Could someone explain why you couldn’t just checkpoint the model before responding to the query (assuming the model context changes as a result of the query, otherwise unnecessary), run the query, and then have another model evaluate if the response was in compliance with the original rules. If it was not, roll back to the checkpoint. Otherwise, proceed. You couldn’t prompt inject the regulator as it is not taking us…
That is a smart idea. I'm curious about the answers to this.
Re: Prompt injection: what’s the worst that can happen?
#127Could someone explain why you couldn’t just checkpoint the model before responding to the query (assuming the model context changes as a result of the query, otherwise unnecessary), run the query, and then have another model evaluate if the response was in compliance with the original rules. If it was not, roll back to the checkpoint. Otherwise, proceed. You couldn’t prompt inject the regulator as it is not taking us…
[deleted]
Re: Prompt injection: what’s the worst that can happen?
#128Earlier quoted context omitted.
Yet, even with the current models, with no special tokens, it is relatively straightforward to construct relatively stable defense. Given that the ongoing attack can be detected, tagged and added to a learned filter, this gets a lot less problematic, in practice. Even a simple limit of interactions is effective enough. Here's a practical prompt defense / CTTF that I've made. With five steps of a dialogue limit (per d…
If/when a breach does happen, are you going to shrug and say “Gee… who would’ve thunk?!”, or are you going to be accountable for the consequences? Is this the kind of defense an LLM service can bet their company on? How do you think a cybersecurity/insurance company will price the risk of a negative fat tailed outcome?
Breaches happen with humans too. Social engineering works. As long as the costs of a breach are not too high, this can simply be a cost of doing business.
Re: Prompt injection: what’s the worst that can happen?
#129Could someone explain why you couldn’t just checkpoint the model before responding to the query (assuming the model context changes as a result of the query, otherwise unnecessary), run the query, and then have another model evaluate if the response was in compliance with the original rules. If it was not, roll back to the checkpoint. Otherwise, proceed. You couldn’t prompt inject the regulator as it is not taking us…
Pretty sure this is the approach bing is taking right now. It'll start to output 'naughty' responses and then they'll be deleted and it'll tell you off.
Re: Prompt injection: what’s the worst that can happen?
#130Could someone explain why you couldn’t just checkpoint the model before responding to the query (assuming the model context changes as a result of the query, otherwise unnecessary), run the query, and then have another model evaluate if the response was in compliance with the original rules. If it was not, roll back to the checkpoint. Otherwise, proceed. You couldn’t prompt inject the regulator as it is not taking us…
The regulator isn't taking user input directly, but it is being exposed to output from a model that was created in response to user input.
So the adversarial attacker needs to come up with original input that will produce output that itself contains a successful prompt injection attack targeting the regulator.
Harder, but I doubt impossible.