Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Multi-modal prompt injection image attacks against GPT-4V

simonwillison.net

Post reply on HN
Print view View original

21–30 of 69 posts

Re: Multi-modal prompt injection image attacks against GPT-4V

#21

Post by simonw »

jcims wrote: post #19
simonw wrote: post #15

Earlier quoted context omitted.

I don't think that's the right way to think about it. It's not running an OCR-style algorithm to export data and then passing that data to the LLM for further processing. Instead, the model itself has been trained on both text and images at the same time. So the image features end up as a weird ball of floating point numbers mixed together with the floating point numbers for the textual representation of the words in…

Ah! Do you think this is like an adjunct or fine-tuning kind of thing or do they retrain GPT 4+ images? That seems like a ginormous task.

When they launched GPT-4 back in March they talked about its image capabilities - I think they trained it into the model from the start, but then spent several extra months on additional safety research, as described in their paper: https://cdn.openai.com/papers/GPTV_System_Card.pdf

Re: Multi-modal prompt injection image attacks against GPT-4V

#22

Post by saberience »

This post broke my brain a little as it goes totally against my mental model of how an LLM is processing images. That is, I don't know why it is processing the content of the image, and then running that through something that then "executes" the text of the image.

So it has some pipeline like -> LLM/OCR Image to Text -> Text to Command Processing. Why is the text description of the image being processed at all and not simply just being output?

As far as I know, we've had engines for detecting sentiment, objects, etc from images for a long time (way before ChatGPT), and they didn't have issues where they would execute a URL found in an image, they would just output the characters...

Can someone who understands this better explain wth is going on here?

Re: Multi-modal prompt injection image attacks against GPT-4V

#23

Post by PeterisP »

simonw wrote: post #16
aleph_minus_one wrote: post #12

Earlier quoted context omitted.

> If you can crack differentiating between "command" tokens and other input tokens, you've cracked prompt injection! In my opinion there exists a rather simple semi-solution (and no, I do not claim to have cracked prompt injection! :-) ): the user is able to mark parts of his input as "trusted" or "untrusted", and the AI is implemented to handle data this way. For example, the user would mark his prompt (that he thor…

> the user is able to mark parts of his input as "trusted" or "untrusted", and the AI is implemented to handle data this way. That's exactly the solution I've been hoping for, but seeing as no-one has even come close to getting that to work yet my current intuition is that it's REALLY hard. Maybe even impossible with current LLM architectures?

There are two main issues in my mind.

First is that the straightforward way to implement any changes to the data structure requires full retraining of the model from scratch, which is very expensive for the large models (I recall reading that GPT-4 training compute cost was sixty million or so) - so it's not something someone would or could do 'for fun' just to test a hypothesis, there needs to be a will to explicitly make the next large model in this manner, because you don't want to make two models at twice the cost.

The second almost all of the training is based on self-supervised data from generic text where there are neither "instructions" or "data", the concepts don't even make sense while you're training on a book or wikipedia article. So all the pretraining would learn to ignore that out-of-band marker, and then you'd have only the RLHF instruction data to try and make some use of it, and you'd need expensive manual review to make this training data - the current RLHF data is much less intensive to annotate because it effectively has just "good"/"bad" signal, and even that can be gathered for free from users; however, this training data would need someone to explicitly annotate what is the prompt and what is data when a user submits a text "please rewrite the grammar for the following paragraph: [... some data copied from internet ..]", you can't magically get that metadata, and it's expensive to annotate a lot of that.

TL;DR; - IMHO it's not difficult, but it is resource consuming, and that fully explains why it's not done.

Re: Multi-modal prompt injection image attacks against GPT-4V

#24

Post by Michelangelo11 »

saberience wrote: post #22

This post broke my brain a little as it goes totally against my mental model of how an LLM is processing images. That is, I don't know why it is processing the content of the image, and then running that through something that then "executes" the text of the image. So it has some pipeline like -> LLM/OCR Image to Text -> Text to Command Processing. Why is the text description of the image being processed at all and n…

Interesting that you say that, because this lines up exactly with my mental models of LLM image processing and LLM functioning in general. One of the defining features of LLMs is that, unlike in regular programming, there's no separation between data and instructions -- they're one and the same -- and that's what we're seeing here. Also, LLMs tend to focus more heavily on the instructions they see last (because of limitations in the attention mechanism, I think? But IIRC nobody has come up with a full explanation yet), and that's what we're seeing here too.

Re: Multi-modal prompt injection image attacks against GPT-4V

#25

Post by simonw »

saberience wrote: post #22

This post broke my brain a little as it goes totally against my mental model of how an LLM is processing images. That is, I don't know why it is processing the content of the image, and then running that through something that then "executes" the text of the image. So it has some pipeline like -> LLM/OCR Image to Text -> Text to Command Processing. Why is the text description of the image being processed at all and n…

See my comment here: https://news.ycombinator.com/item?id=37880515

I don't think it's accurate to think of the image processing as a separate pipeline here, similar to OCR on object detection.

My understanding is that the image is treated the same as the text - it's all input into the weird giant matrices of numbers that makes up the model.

GPT4-V isn't GPT4 plus tools for analyzing images: it's GPT4 with image processing built right into the model itself.

Is it even right to call it a "Large Language Model" at this point? I'm not sure.

Re: Multi-modal prompt injection image attacks against GPT-4V

#26

Post by Michelangelo11 »

simonw wrote: post #7
magicalhippo wrote: post #6

Not my field but I'd assume they had encoded tokens extracted from an image differently than prompt tokens, so they wouldn't get interpreted as a command. Why aren't they doing this? Or are they, it's just failing?

I don't think anyone has figured out how to implement this kind of LLM system in that way. If you can crack differentiating between "command" tokens and other input tokens, you've cracked prompt injection! It's been 13 months since we started discussing this vulnerability and I've not seen any evidence of anyone cracking this yet.

Right, exactly. As I said in another comment on this thread, one of the defining features of LLMs is that there's no separation between data and instructions -- and I don't think there's any way to fix that within the current LLM paradigm.

Re: Multi-modal prompt injection image attacks against GPT-4V

#27

Post by Filligree »

simonw wrote: post #16
aleph_minus_one wrote: post #12

Earlier quoted context omitted.

> If you can crack differentiating between "command" tokens and other input tokens, you've cracked prompt injection! In my opinion there exists a rather simple semi-solution (and no, I do not claim to have cracked prompt injection! :-) ): the user is able to mark parts of his input as "trusted" or "untrusted", and the AI is implemented to handle data this way. For example, the user would mark his prompt (that he thor…

> the user is able to mark parts of his input as "trusted" or "untrusted", and the AI is implemented to handle data this way. That's exactly the solution I've been hoping for, but seeing as no-one has even come close to getting that to work yet my current intuition is that it's REALLY hard. Maybe even impossible with current LLM architectures?

If you use the GPT API you can just define your own system prompt… but it’s not as useful as you’d hope. The API isn’t particularly censored to begin with; that’s mostly ChatGPT

Re: Multi-modal prompt injection image attacks against GPT-4V

#28

Post by Michelangelo11 »

simonw wrote: post #25
saberience wrote: post #22

This post broke my brain a little as it goes totally against my mental model of how an LLM is processing images. That is, I don't know why it is processing the content of the image, and then running that through something that then "executes" the text of the image. So it has some pipeline like -> LLM/OCR Image to Text -> Text to Command Processing. Why is the text description of the image being processed at all and n…

See my comment here: https://news.ycombinator.com/item?id=37880515 I don't think it's accurate to think of the image processing as a separate pipeline here, similar to OCR on object detection. My understanding is that the image is treated the same as the text - it's all input into the weird giant matrices of numbers that makes up the model. GPT4-V isn't GPT4 plus tools for analyzing images: it's GPT4 with image proce…

Large Comprehension Model :)

Re: Multi-modal prompt injection image attacks against GPT-4V

#29

Post by dragonwriter »

simonw wrote: post #7
magicalhippo wrote: post #6

Not my field but I'd assume they had encoded tokens extracted from an image differently than prompt tokens, so they wouldn't get interpreted as a command. Why aren't they doing this? Or are they, it's just failing?

I don't think anyone has figured out how to implement this kind of LLM system in that way. If you can crack differentiating between "command" tokens and other input tokens, you've cracked prompt injection! It's been 13 months since we started discussing this vulnerability and I've not seen any evidence of anyone cracking this yet.

> If you can crack differentiating between "command" tokens and other input tokens, you've cracked prompt injection!

Isn't this simple? Have two different tokenizer with non-overlapping output ranges, one for “trusted commands” and one for “user input”, do training with various mixes of commands and input with the appropriate reactions to each, and then have the trusted command fixed (or selectable from a small set of preselected options/payterns based on interaction style [chat might used trusted tokens to indicate which participant each piece of the exchange is from, as well as for the internal system prompt, for instance]) for external users and allow them to freely specify only input that gets sent to the “untrusted input” tokenizer. Even if you allow pretokenized input, you can sanitize it to accept only tokens from the untrusted space.

Re: Multi-modal prompt injection image attacks against GPT-4V

#30

Post by saberience »

Michelangelo11 wrote: post #24
saberience wrote: post #22

This post broke my brain a little as it goes totally against my mental model of how an LLM is processing images. That is, I don't know why it is processing the content of the image, and then running that through something that then "executes" the text of the image. So it has some pipeline like -> LLM/OCR Image to Text -> Text to Command Processing. Why is the text description of the image being processed at all and n…

Interesting that you say that, because this lines up exactly with my mental models of LLM image processing and LLM functioning in general. One of the defining features of LLMs is that, unlike in regular programming, there's no separation between data and instructions -- they're one and the same -- and that's what we're seeing here. Also, LLMs tend to focus more heavily on the instructions they see last (because of li…

So the original release of ChatGPT would call a URL you entered in the chat? I don't recall it acting that way. In the linked post ChatGPT literally goes to a URL from an image... i.e. it calls something like curl with that URL. This definitely seems like something adjacent to an LLM.
Post reply on HN