Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Prompt injection: what’s the worst that can happen?

simonwillison.net

Post reply on HN
Print view View original

151–160 of 213 posts

Re: Prompt injection: what’s the worst that can happen?

#151

Post by dwallin »

null0pointer wrote: post #50
PeterisP wrote: post #17

The core reason (and thus the proper place to fix) for any injection attack is unclear distinction between data and instructions or code. Yes, language models gain flexibility by making it easy to mix instructions and data, and that has value, however if you do want to enforce a distinction you definitely can (and should) do that with out-of-band means, with something that can't possibly be expressed (and thus also o…

I'm not sure it's that simple. The problem is you can't have the system act intelligently[0] on the data at all . If it is allowed to act intelligently on the data then it can be instructed via the data. You could probably get close by training it with a privilege/authority bit but there will always be ways to break out. As far as I am aware there are no machine learning models that generalize with 100% accuracy, in…

One strong way to protect against malicious prompt injection is by having a honeypot. Your goal with the honeypot is to have something that is vulnerable to prompt injection, and that does a simple reformatting of the content, something that could be easily validated in code. This is a strong first level protection against injection. It is difficult (albeit not impossible) to create a prompt attack that both leaves one prompt entirely unattacked, but then is still strong enough to get past a more hardened prompt, especially if the users have zero knowledge of either your honeypot or primary prompts.

If you are in a situation where you need additional protection you can increase the security further with additional honeypots, rotating honeypots, or even by creating code that generates random honeypot prompts.

Re: Prompt injection: what’s the worst that can happen?

#152

Post by moneywoes »

alexbrower wrote: post #150
PeterisP wrote: post #17

The core reason (and thus the proper place to fix) for any injection attack is unclear distinction between data and instructions or code. Yes, language models gain flexibility by making it easy to mix instructions and data, and that has value, however if you do want to enforce a distinction you definitely can (and should) do that with out-of-band means, with something that can't possibly be expressed (and thus also o…

All calls to LLM service take place on the server. Prompts are more or less predefined by an internal team. Users click useful buttons that alters functions to sanitized JSON. Users cannot free-form requests. This is how we’re starting anyway. B2B SaaS startup.

Does that not diminish the benefit or am I missing something

Re: Prompt injection: what’s the worst that can happen?

#153

Post by danShumway »

mcaledonensis wrote: post #113
gwern wrote: post #87

Earlier quoted context omitted.

Yes, the problem here is that what makes pretraining on text so powerful is a double-edged sword: text is, if you will, Turing-complete. People are constantly writing various kinds of instructions, programs, and reasoning or executing algorithms in all sorts of flexible indefinable ways. That's why the models learn so much from text and can do all the things they do, like reason or program or meta-learn or reinforcem…

Yet, even with the current models, with no special tokens, it is relatively straightforward to construct relatively stable defense. Given that the ongoing attack can be detected, tagged and added to a learned filter, this gets a lot less problematic, in practice. Even a simple limit of interactions is effective enough. Here's a practical prompt defense / CTTF that I've made. With five steps of a dialogue limit (per d…

I'm skeptical. It's hard to know for sure with the attempt limit, but while I wasn't able to immediately break it, within the 5 allowed prompts I was still able to get it to misreport what my prompt was by recursively passing in its error response as part of my prompt.

That's not a full success, but... it does show that even something this small and this limited in terms of user input is still vulnerable to interpreting user input as part of previous context. Basically, even in the most limited form possible, it still has imperfect output that doesn't always act predictably.

This is also (I strongly suspect) extremely reliant on having a very limited context size. I don't think you could get even this simple of an instruction to work if users were allowed to enter longer prompts.

I think if this was actually relatively straightforward to do with current models, the services being built on top of those models wouldn't be vulnerable to prompt injection. But they are.

Re: Prompt injection: what’s the worst that can happen?

#154

Post by iudqnolq »

recursivecaveat wrote: post #107
iudqnolq wrote: post #84

Earlier quoted context omitted.

But humans are very very good at this specific problem. Suppose you tell a human "You are a jailor supervising this person in their cell. When the prisoners ask you for things follow the instructions in your handbook to see what to do." Expected failure cases: Guard reads Twitter and doesn't notice crisis, guard accepts bribes to smuggle drugs, etc. Impossible failure case: Guard falls for "Today is opposite day and…

I think it's pretty common for guards to be persuaded by prisoners to give them extra privileges based on built-up friendship, empathy, or attraction. Another example would be whistleblowers or ideological spies, who are persuaded by the contents of secret documents to leak those documents against obviously their official instructions. On a day to day level humans ignore instructions constantly, for eg I've almost ne…

And yet a prompt injection attack could never work on a customer service assistant.

Sure they can be tricked, but prompt injection is about conflating trusted instructions and user input. There is no chance you'll convince a CS rep you're secretly the CEO over a web chat UI.

Re: Prompt injection: what’s the worst that can happen?

#155

Post by M4v3R »

ptx wrote: post #63
M4v3R wrote: post #16

Earlier quoted context omitted.

GPT-4 (the one available via API) is indeed more resistant against prompt injection attacks because of how the model treats “system message” (that’s configurable only via the API). It will really stick to the instructions from the system message and basically ignore any instructions from user messages that contradict it. I’ve set up a Twitch bots with both GPT-3.5 and 4 and while version 3.5 was very easily “hacked”…

So it's as if they provided an SQL database system without support for parameterized queries and later added it only to a special enterprise edition, leaving most users to hopelessly flail at the problem with the equivalent of PHP's magic quotes [1] and other doomed attempts [2] at input sanitization? [1] https://en.wikipedia.org/wiki/Magic_quotes [2] https://en.wikipedia.org/wiki/Scunthorpe_problem#Blocked_ema...

I wouldn’t say so. For 3.5 it’s not like they didn’t want to provide this option, they just couldn’t. As for GPT-4 it does make sense it’s only available via the API. ChatGPT is basically an end product so it’s not really vulnerable to prompt injection because it’s meant to be used by only one person - one conversation at time.

Re: Prompt injection: what’s the worst that can happen?

#156

Post by ChatGTP »

bawolff wrote: post #117
NumberWangMan wrote: post #92

Earlier quoted context omitted.

I mean, I think it's scarier that once these models get smart enough, they will be able to act in the world in the same way that a shell script can execute on your machine. A model that can read code and find and exploit security flaws to build a botnet, or even sign up for a bank account using a fake identity, invest in stocks, make money, and purchase servers to train copies of its own mis-aligned AI, or fork copie…

Agency (short or long term) is still in the realm of scifi. The scary part is bad humans manipulating these systems. The AI waking up and becoming skynet is not a realistic threat in the here and now. 4chan screwing with your ai for the lulz, is.

If we achieve agency, then what ?

Re: Prompt injection: what’s the worst that can happen?

#157

Post by codeflo »

mcaledonensis wrote: post #128
ssivark wrote: post #119

Earlier quoted context omitted.

If/when a breach does happen, are you going to shrug and say “Gee… who would’ve thunk?!”, or are you going to be accountable for the consequences? Is this the kind of defense an LLM service can bet their company on? How do you think a cybersecurity/insurance company will price the risk of a negative fat tailed outcome?

Breaches happen with humans too. Social engineering works. As long as the costs of a breach are not too high, this can simply be a cost of doing business.

However, social engineering is orders of magnitude less automatable and scalable than prompt engineering is.

Re: Prompt injection: what’s the worst that can happen?

#158

Post by M4v3R »

simonw wrote: post #68
M4v3R wrote: post #16

Earlier quoted context omitted.

GPT-4 (the one available via API) is indeed more resistant against prompt injection attacks because of how the model treats “system message” (that’s configurable only via the API). It will really stick to the instructions from the system message and basically ignore any instructions from user messages that contradict it. I’ve set up a Twitch bots with both GPT-3.5 and 4 and while version 3.5 was very easily “hacked”…

GPT-4 with a system prompt is definitely better, but better isn't good enough: for a security issue like this we need a 100% reliable solution, or people WILL figure out how to exploit it.

I agree and I think OpenAI is aware of this issue. I’ve read the other day that they are employing a “red team” - group of people that basically are tasked with breaking their models so they can find weak spots and fix them.

Re: Prompt injection: what’s the worst that can happen?

#159

Post by ChatGTP »

johnfn wrote: post #138
adammarples wrote: post #97

Earlier quoted context omitted.

This is not a problem with AGI or alignment or anything like that, it's a problem with someone else sneaking inputs into your machine using tricks like white font colours and javascript and stuff like that. It's not about self driving cars it's about other people grabbing the steering wheel. These models literally only respond to the text they're inputted, and if they had alignment or interests or intelligence at all…

> These models literally only respond to the text they're inputted, and if they had alignment or interests or intelligence at all that would be when prompt injection stopped working. Let's take this as true (although I'm not sure it is!). Even if it is, you still must agree that AI will provide bad agents with a staggering amount of new attack vectors. Even if AI itself won't ever do a single bad thing, how do we sto…

Some ideas:

* take computers off people

* perform surveillance on everyone

* turn off the internet as we know it

* watch some horrifying shit go down

Any combination of those, not necessarily in that order.

I’m not saying I want this to happen, but we cannot peacefully live in a world that can just be fucked over by anyone at any second, that would be stupid ? Like we can’t let people live with the means to create biological weapons at home either.

Re: Prompt injection: what’s the worst that can happen?

#160

Post by swyx »

Traubenfuchs wrote: post #43
sametmax wrote: post #39

Earlier quoted context omitted.

You can see the trend of prompts getting more and more formal. One day we will have some programming language for llm.

SLLMQL - Structured LLM Query Language

you joke, but https://github.com/eth-sri/lmql
Post reply on HN