Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Multi-modal prompt injection image attacks against GPT-4V

simonwillison.net

Post reply on HN
Print view View original

41–50 of 69 posts

Re: Multi-modal prompt injection image attacks against GPT-4V

#41

Post by Michelangelo11 »

PeterisP wrote: post #23
simonw wrote: post #16

Earlier quoted context omitted.

> the user is able to mark parts of his input as "trusted" or "untrusted", and the AI is implemented to handle data this way. That's exactly the solution I've been hoping for, but seeing as no-one has even come close to getting that to work yet my current intuition is that it's REALLY hard. Maybe even impossible with current LLM architectures?

There are two main issues in my mind. First is that the straightforward way to implement any changes to the data structure requires full retraining of the model from scratch, which is very expensive for the large models (I recall reading that GPT-4 training compute cost was sixty million or so) - so it's not something someone would or could do 'for fun' just to test a hypothesis, there needs to be a will to explicitl…

But I still don't see how that form of RLHF could be fully, mechanistically reliable, which is what you'd need here. Current RLHF drastically lowers the likelihood of the LLM generating objectionable content, but it doesn't drop it to zero, because LLMs need to be highly pliant and suggestible to be useful and that leaves a window of opportunity for bad actors. How would this be any different?

Re: Multi-modal prompt injection image attacks against GPT-4V

#42

Post by aleph_minus_one »

simonw wrote: post #36
PeterisP wrote: post #23

Earlier quoted context omitted.

There are two main issues in my mind. First is that the straightforward way to implement any changes to the data structure requires full retraining of the model from scratch, which is very expensive for the large models (I recall reading that GPT-4 training compute cost was sixty million or so) - so it's not something someone would or could do 'for fun' just to test a hypothesis, there needs to be a will to explicitl…

Anyone who can build an API to an LLM that was provably secure against prompt injection would have a license to print money right now. I don't think the lack of a solution is because it's too expensive. It's because we don't know how to do it yet.

> Anyone who can build an API to an LLM that was provably secure against prompt injection would have a license to print money right now.

Well, if you could create software that is provably secure against at least a large class of cyberattacks, you could also make quite some money, since I know that at least the finance industry would be willing to pay a lot of money for it (source: in the past, I had an opportunity to talk someone who is very knowledgable about software in the finance sector, and I asked him some questions about exactly this topic).

Thus: we have not solved the "cyberinjection" problem for completely conventional software for decades. So I don't believe that we suddenly have the capabilities to solve this problem for LLMs.

But: As I pointed out, people have come up with ideas that make cyberattacks much harder. Here, I can imagine that their ideas could be used to make prompt injections a lot more complicated.

Additionally, we know from decades of experience that making insecure software secure is nigh impossible. Instead, one has to architect around security properties from beginning. The same does, in my opinion, also hold for LLMs and prompt injection.

Re: Multi-modal prompt injection image attacks against GPT-4V

#43

Post by GaggiX »

simonw wrote: post #21
jcims wrote: post #19

Earlier quoted context omitted.

Ah! Do you think this is like an adjunct or fine-tuning kind of thing or do they retrain GPT 4+ images? That seems like a ginormous task.

When they launched GPT-4 back in March they talked about its image capabilities - I think they trained it into the model from the start, but then spent several extra months on additional safety research, as described in their paper: https://cdn.openai.com/papers/GPTV_System_Card.pdf

I don't think they have trained the model from scratch on images too, but more like Flamingo or similar methods, and the LLM model is at most finetuned but not trained from scratch.

If the model was trained from scratch then I would expect it to be able to develop new emergent behavior like being much better at ascii art, SVG, OpenSCAD scripting at least in the case the model was trained on the usual masked objective but not only on text tokens but image tokens too.

Re: Multi-modal prompt injection image attacks against GPT-4V

#44

Post by magicalhippo »

simonw wrote: post #33
dragonwriter wrote: post #29

Earlier quoted context omitted.

> If you can crack differentiating between "command" tokens and other input tokens, you've cracked prompt injection! Isn't this simple? Have two different tokenizer with non-overlapping output ranges, one for “trusted commands” and one for “user input”, do training with various mixes of commands and input with the appropriate reactions to each, and then have the trusted command fixed (or selectable from a small set o…

Definitely not simple. If it was simple we would have solved it already!

Do you have some references (articles, blogs or similar) that illuminates why the trivial solutions won't work?

Clearly I know enough to come up with a trivial solution, but not enough to immediately see why it won't work.

Re: Multi-modal prompt injection image attacks against GPT-4V

#45

Post by simonw »

magicalhippo wrote: post #44
simonw wrote: post #33

Earlier quoted context omitted.

Definitely not simple. If it was simple we would have solved it already!

Do you have some references (articles, blogs or similar) that illuminates why the trivial solutions won't work? Clearly I know enough to come up with a trivial solution, but not enough to immediately see why it won't work.

I've written a lot about this myself: https://simonwillison.net/series/prompt-injection/

Re: Multi-modal prompt injection image attacks against GPT-4V

#46

Post by famouswaffles »

sebzim4500 wrote: post #20
jgalt212 wrote: post #10

Earlier quoted context omitted.

I feel differently. I think Simon's wasting his talents on an evolutionary dead end. If transformers and LLMs held the answer, self-driving cars would be getting better.

What do LLMs have to do with self-driving cars? Also they are getting better, just slowly.

Guess he's expecting something like this to happen faster?

https://wayve.ai/thinking/lingo-natural-language-autonomous-...

Re: Multi-modal prompt injection image attacks against GPT-4V

#47

Post by dragonwriter »

simonw wrote: post #39
dragonwriter wrote: post #38

Earlier quoted context omitted.

Always love the “there are no easy next steps from whwre we currently are, because if they were someone would already have taken them” argument.

Because it's true, right? I'm not saying there aren't any HARD steps. Every time I talk about prompt injection the same "surely there's an easy fix: ...." comments show up. I'm just trying to shortcut having to argue against them all again.

> Because it's true, right?

No, while sometimes there are barriers that take a giant leap to cross, progress tends to involveots of steps that were simple, even if time consuming to execute, once the preceding pieces were in place.

Re: Multi-modal prompt injection image attacks against GPT-4V

#48

Post by mk_stjames »

simonw wrote: post #15
jcims wrote: post #14

Earlier quoted context omitted.

I’m super curious about the data structure that’s being extracted from the image and fed to GPT. I asked it to estimate the size of a figurine based on a ruler in the image, and it nailed it. https://imgur.com/a/gsiID4v At first I just thought it was able taking an array of objects identified in the photo with bounding box coordinates and then just figuring out what the answer is. But then i asked it to describe Worf…

I don't think that's the right way to think about it. It's not running an OCR-style algorithm to export data and then passing that data to the LLM for further processing. Instead, the model itself has been trained on both text and images at the same time. So the image features end up as a weird ball of floating point numbers mixed together with the floating point numbers for the textual representation of the words in…

The entire GPT4 model was not retrained to utilize images, instead I think it is much more like show in [1] where an extra layer(s) are learned that take another pretrained image model and map that model to the latent space of the LLM. In [1] this was just a single linear layer. I found it pretty shocking how simple it was. I believe OpenAI is doing something similar.

[1] https://arxiv.org/pdf/2304.10592.pdf

Re: Multi-modal prompt injection image attacks against GPT-4V

#49

Post by simonw »

dragonwriter wrote: post #47
simonw wrote: post #39

Earlier quoted context omitted.

Because it's true, right? I'm not saying there aren't any HARD steps. Every time I talk about prompt injection the same "surely there's an easy fix: ...." comments show up. I'm just trying to shortcut having to argue against them all again.

> Because it's true, right? No, while sometimes there are barriers that take a giant leap to cross, progress tends to involveots of steps that were simple, even if time consuming to execute, once the preceding pieces were in place.

If there genuinely is a simple solution to prompt injection that's been eluding us all for 13 months I cannot wait to learn what it is!

Re: Multi-modal prompt injection image attacks against GPT-4V

#50

Post by dragonwriter »

simonw wrote: post #16
aleph_minus_one wrote: post #12

Earlier quoted context omitted.

> If you can crack differentiating between "command" tokens and other input tokens, you've cracked prompt injection! In my opinion there exists a rather simple semi-solution (and no, I do not claim to have cracked prompt injection! :-) ): the user is able to mark parts of his input as "trusted" or "untrusted", and the AI is implemented to handle data this way. For example, the user would mark his prompt (that he thor…

> the user is able to mark parts of his input as "trusted" or "untrusted", and the AI is implemented to handle data this way. That's exactly the solution I've been hoping for, but seeing as no-one has even come close to getting that to work yet my current intuition is that it's REALLY hard. Maybe even impossible with current LLM architectures?

I don't think its an architecture (on the high level; there are probably architecture details like the specifics of the tokenizer that need adaptation in a straightforward way) problem, I think its a training/dataset generation/labelling problem.

That is, I think it is both simple and incredibly expensive/arduous, because the masses of free data to train models on doesn't come with a command-space/untrusted-input-space dichotomy, that’s something you have to impose onto the training data, with appropriately differentiated completions.

(On top of the fact that it likely requires de novo model training, which is itself expensive.)

Post reply on HN