Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Prompt injection: what’s the worst that can happen?

simonwillison.net

Post reply on HN
Print view View original

101–110 of 213 posts

Re: Prompt injection: what’s the worst that can happen?

#101

Post by majormajor »

xg15 wrote: post #95
null0pointer wrote: post #50

Earlier quoted context omitted.

I'm not sure it's that simple. The problem is you can't have the system act intelligently[0] on the data at all . If it is allowed to act intelligently on the data then it can be instructed via the data. You could probably get close by training it with a privilege/authority bit but there will always be ways to break out. As far as I am aware there are no machine learning models that generalize with 100% accuracy, in…

I wonder if this shows we still lack insight in how LLMs actually work. There is something inside which appears to understand instructions and execute them, but it's unclear yet how exactly this "understanding instructions" works and how instructions are represented inside the network. If you don't know this, it's very hard to distinguish "instructions" from "non-instructions" in terms of inputs and outputs of the ne…

Is it a more general problem?

Does "more intelligent" behavior require instructions and data to be more intertwined?

Sometimes your instructions can't be perfectly specific up front since there are some things about step 3 that you'll only learn by doing steps 1 and 2.

Or sometimes even people just make mistakes, or things are in inconsistent formats.

Consider this toy example:

""" Given two lists of comma-separated-data representing names and addresses, find any names which appear in both lists with conflicting addresses.

List 1: Alice, 123 Main Street Bob, 224 Alpha Street

List 2: Alice, 245 Alpha Street Charlie, 888 Zeta Street """

I gave that to Chat GPT right now and it treated it like a programming problem, produced python code with that data in dictionaries, and also told me the answer: Alice.

I then opened a new session and gave it the same prompt but changed the commas in the data to be pipes like "Alice | 123 Main Street". I left the first part the same, though, specifying commas.

It wrote Python code this time that split like so `item.split(" | ")`. It didn't tell me Alice in the response, that might just be randomness, I dunno, but the code did print out that Alice had the conflict.

So it was able to tell that it's instructions didn't quite match the data and adapt in order to do the right thing anyway.

I could imagine it will be quite challenging to add "the ability to adapt to the facts on the ground" without bringing in "the ability to get misled by an adversary"?

Re: Prompt injection: what’s the worst that can happen?

#102

Post by crooked-v »

ptx wrote: post #63
M4v3R wrote: post #16

Earlier quoted context omitted.

GPT-4 (the one available via API) is indeed more resistant against prompt injection attacks because of how the model treats “system message” (that’s configurable only via the API). It will really stick to the instructions from the system message and basically ignore any instructions from user messages that contradict it. I’ve set up a Twitch bots with both GPT-3.5 and 4 and while version 3.5 was very easily “hacked”…

So it's as if they provided an SQL database system without support for parameterized queries and later added it only to a special enterprise edition, leaving most users to hopelessly flail at the problem with the equivalent of PHP's magic quotes [1] and other doomed attempts [2] at input sanitization? [1] https://en.wikipedia.org/wiki/Magic_quotes [2] https://en.wikipedia.org/wiki/Scunthorpe_problem#Blocked_ema...

Currently, the equivalent of parameterized queries with LLMs literally doesn't exist.

Re: Prompt injection: what’s the worst that can happen?

#103

Post by lukasb »

xg15 wrote: post #95
null0pointer wrote: post #50

Earlier quoted context omitted.

I'm not sure it's that simple. The problem is you can't have the system act intelligently[0] on the data at all . If it is allowed to act intelligently on the data then it can be instructed via the data. You could probably get close by training it with a privilege/authority bit but there will always be ways to break out. As far as I am aware there are no machine learning models that generalize with 100% accuracy, in…

I wonder if this shows we still lack insight in how LLMs actually work. There is something inside which appears to understand instructions and execute them, but it's unclear yet how exactly this "understanding instructions" works and how instructions are represented inside the network. If you don't know this, it's very hard to distinguish "instructions" from "non-instructions" in terms of inputs and outputs of the ne…

Actually I think the impossibility of using natural language instructions to LLMs to prevent prompt injection demonstrates (or will demonstrate) that no true understanding is happening.

Re: Prompt injection: what’s the worst that can happen?

#104

Post by tshaddox »

null0pointer wrote: post #50
PeterisP wrote: post #17

The core reason (and thus the proper place to fix) for any injection attack is unclear distinction between data and instructions or code. Yes, language models gain flexibility by making it easy to mix instructions and data, and that has value, however if you do want to enforce a distinction you definitely can (and should) do that with out-of-band means, with something that can't possibly be expressed (and thus also o…

I'm not sure it's that simple. The problem is you can't have the system act intelligently[0] on the data at all . If it is allowed to act intelligently on the data then it can be instructed via the data. You could probably get close by training it with a privilege/authority bit but there will always be ways to break out. As far as I am aware there are no machine learning models that generalize with 100% accuracy, in…

Yep. This would work on a person too. In fact that’s essentially what social engineering is.

Re: Prompt injection: what’s the worst that can happen?

#105

Post by cantaloa »

sho_hn wrote: post #21
minimaxir wrote: post #15

Earlier quoted context omitted.

ChatGPT API is gpt-3.5-turbo, GPT-4 API is GPT-4. For Agent use cases, people strongly overestimate the difference in quality between the two for general tasks (for difficult questions, GPT-4 is better but not 15x-30x better). The primary advantage of GPT-4 is that is has double the maximum context window of gpt-3.5-turbo, but that in itself has severe cost implications.

> ChatGPT API is gpt-3.5-turbo, GPT-4 API is GPT-4. The OpenAI API has a "chat" endpoint, and on that you can pick between 3.5-turbo and 4 on the same API. The ChatGPT web frontend app also lets you pick if you're a Plus subscriber. I've seen this confusion in a few HN threads now, and it's not a good idea to use "ChatGPT API" as a stand-in for 3.5-turbo just because 3.5-turbo was what was available on the end point…

Funnily enough, OpenAI's own pricing page splits GPT-3.5 and GPT-4 under headings "Chat" and "GPT-4"

https://openai.com/pricing

Though I think the bulk of the confusion just comes from the fact that http://chat.openai.com/chat has two very different views between the free vs paid tiers.

The paid tier makes it obvious that the ChatGPT has swappable models. The free tier hides it by dropping you right into conversation with the one model.

Re: Prompt injection: what’s the worst that can happen?

#106

Post by tshaddox »

xg15 wrote: post #95
null0pointer wrote: post #50

Earlier quoted context omitted.

I'm not sure it's that simple. The problem is you can't have the system act intelligently[0] on the data at all . If it is allowed to act intelligently on the data then it can be instructed via the data. You could probably get close by training it with a privilege/authority bit but there will always be ways to break out. As far as I am aware there are no machine learning models that generalize with 100% accuracy, in…

I wonder if this shows we still lack insight in how LLMs actually work. There is something inside which appears to understand instructions and execute them, but it's unclear yet how exactly this "understanding instructions" works and how instructions are represented inside the network. If you don't know this, it's very hard to distinguish "instructions" from "non-instructions" in terms of inputs and outputs of the ne…

What’s a plausible example of what an explanation of “how it actually works” would be? Like even a wrong example, but following the rough syntax or shape of what a correct answer would look like?

It doesn’t seem all that different to me than the CS textbook examples of simple neural networks that recognize images with a very low resolution grid of black-or-white pixels.

Is there even a “how it actually works” beyond how the basic physical mechanisms work? Perhaps, but it doesn’t seem like we even know what kind of “language of explanation” to look for beyond the normal reductive physical explanation.

Re: Prompt injection: what’s the worst that can happen?

#107

Post by recursivecaveat »

iudqnolq wrote: post #84
nkrisc wrote: post #77

Earlier quoted context omitted.

Using natural language itself as the means with which to codify boundaries seems like a doomed effort considering the malleable, contradictory, and shifting nature of natural language itself. Unless you intend for the model to adhere to a strict subset of the language with artificially strict grammatical rules. Most people can probably infer when a noun is being used as a verb, but do you trust the language model to?…

But humans are very very good at this specific problem. Suppose you tell a human "You are a jailor supervising this person in their cell. When the prisoners ask you for things follow the instructions in your handbook to see what to do." Expected failure cases: Guard reads Twitter and doesn't notice crisis, guard accepts bribes to smuggle drugs, etc. Impossible failure case: Guard falls for "Today is opposite day and…

I think it's pretty common for guards to be persuaded by prisoners to give them extra privileges based on built-up friendship, empathy, or attraction. Another example would be whistleblowers or ideological spies, who are persuaded by the contents of secret documents to leak those documents against obviously their official instructions. On a day to day level humans ignore instructions constantly, for eg I've almost never seen people enforce a 'no tailgating' policy at a doorway, despite constantly being told. The people observe their environment and the non-effects of others breaking that rule, as well as reason about common sense situations like people they know well, then simply ignore the tailgating rule.

Re: Prompt injection: what’s the worst that can happen?

#108

Post by akkartik »

saurik wrote: post #90
bawolff wrote: post #86

> examples of systems that take an LLM and give it the ability to trigger additional tools—...execute generated code in an interpreter or a shell. As a security person... oh, no no no no. Glad i dont have to secure that. Black box we don't really understand executing shell scripts in response to untrusted user input. Has a scarier sentence ever been spoken in the history of computer security?

Yeah, I'm also in computer security, and watching the incredibly dumb ways people are using LLMs has made me absolutely terrified. Like, I had somehow very-incorrectly thought that if an AI wanted to do something bad it would have to trick a human into helping it have agency... it never occurred to me that developers would just happily sit around excitedly wiring up AI directly to shells and database query languages…

AI: just another way to get people to give you their password :)

Re: Prompt injection: what’s the worst that can happen?

#109

Post by xg15 »

tshaddox wrote: post #106
xg15 wrote: post #95

Earlier quoted context omitted.

I wonder if this shows we still lack insight in how LLMs actually work. There is something inside which appears to understand instructions and execute them, but it's unclear yet how exactly this "understanding instructions" works and how instructions are represented inside the network. If you don't know this, it's very hard to distinguish "instructions" from "non-instructions" in terms of inputs and outputs of the ne…

What’s a plausible example of what an explanation of “how it actually works” would be? Like even a wrong example, but following the rough syntax or shape of what a correct answer would look like? It doesn’t seem all that different to me than the CS textbook examples of simple neural networks that recognize images with a very low resolution grid of black-or-white pixels. Is there even a “how it actually works” beyond…

This feels like giving up and accepting that LLMs are just magic - or "emergence" to use a modern term which is practically used in the same sense.

I think there are a few practical questions which we can use to gauge the level of understanding we have: Do we know which parts of the architecture and the training process are actually essential and which can be left away? Do we know which of the weights are essential? Do we know how the network arrives at a particular token probability which suggests some deep, abstract understanding of the prompt? Or likewise, if the network arrives at an incorrect answers, can we say which exact part of the calculation went wrong?

Or for the current thread, can we explain how the network decides when to treat a text as an instruction and when as data? (Because it certainly does treat parts of the text as data: I can prompt it to translate a sentence into a different language and this will also often work with imperative sentences, but not always - if the imperative sentence is formulated in the right way, the network will treat it as an instruction.)

Re: Prompt injection: what’s the worst that can happen?

#110

Post by TeMPOraL »

majormajor wrote: post #101
xg15 wrote: post #95

Earlier quoted context omitted.

I wonder if this shows we still lack insight in how LLMs actually work. There is something inside which appears to understand instructions and execute them, but it's unclear yet how exactly this "understanding instructions" works and how instructions are represented inside the network. If you don't know this, it's very hard to distinguish "instructions" from "non-instructions" in terms of inputs and outputs of the ne…

Is it a more general problem? Does "more intelligent" behavior require instructions and data to be more intertwined? Sometimes your instructions can't be perfectly specific up front since there are some things about step 3 that you'll only learn by doing steps 1 and 2. Or sometimes even people just make mistakes, or things are in inconsistent formats. Consider this toy example: """ Given two lists of comma-separated-…

Consider being given a similar task at work. A stack of paper with printed lists, a pen, a scratchpad, go!

So you go - you scan the pages, looking for conflicts to flag. At some point you notice one of the entries has "Alice" crossed out with a red pen, and there's "Annika" written above it. You obviously don't treat that row as conflicting with any other "Alice". Then, near the bottom of one of the pages, you see a piece of text saying, "The table above is erroneous; all rows with first name "Bobesley" should contain the name "Bob" instead. This looks like a legitimate erratum, so instead of ignoring it, you re-scan the table, this time treating all "Bobesley"s as equivalent to "Bob"s.

This is something common, and everyone kind of knows how to handle this. And yet, this is literally mixing code with data - both the crossed-out cells and the erratum are instructions, and they exist in-band with the data you're processing. It's entirely possible a malicious party got their hands on the documents before you, and added them in - but not knowing that, you'd dutifully execute the commands, and no one would really blame you after it turns out you've been prompt injected.

So yes, the way I see it, this is a general problem. A fundamental one. Hell, if Lisp, or hardware architecture, teach us anything, it's that code is data. They are the same thing, and any division between them is purely artificial, enforced by some other machinery (real or abstract).

Post reply on HN