Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Multi-modal prompt injection image attacks against GPT-4V

simonwillison.net

Post reply on HN
Print view View original

11–20 of 69 posts

Re: Multi-modal prompt injection image attacks against GPT-4V

#11

Post by simonw »

jgalt212 wrote: post #10
Roritharr wrote: post #2

I love simonw content. It's like a day after I saw something interesting on X wrt AI, I can trust that there will be an article a day later by simonw where he tried the things I would have wanted to try and explains them nicely. Thanks for that!

I feel differently. I think Simon's wasting his talents on an evolutionary dead end. If transformers and LLMs held the answer, self-driving cars would be getting better.

As a daily user of LLMs for over a year, I'm confident that they're not a waste of my time.

Even if development were to freeze, they didn't improve at all from this day onwards, and their many monumental flaws (prompt injection, hallucination, inability to reason etc) were never solved, I still think they'd be worth studying and using.

I expect we could still spend years figuring out new capabilities in the models we already have access to today.

Re: Multi-modal prompt injection image attacks against GPT-4V

#12

Post by aleph_minus_one »

simonw wrote: post #7
magicalhippo wrote: post #6

Not my field but I'd assume they had encoded tokens extracted from an image differently than prompt tokens, so they wouldn't get interpreted as a command. Why aren't they doing this? Or are they, it's just failing?

I don't think anyone has figured out how to implement this kind of LLM system in that way. If you can crack differentiating between "command" tokens and other input tokens, you've cracked prompt injection! It's been 13 months since we started discussing this vulnerability and I've not seen any evidence of anyone cracking this yet.

> If you can crack differentiating between "command" tokens and other input tokens, you've cracked prompt injection!

In my opinion there exists a rather simple semi-solution (and no, I do not claim to have cracked prompt injection! :-) ): the user is able to mark parts of his input as "trusted" or "untrusted", and the AI is implemented to handle data this way.

For example, the user would mark his prompt (that he thoroughly thought through) as "trusted", but the image which provides additional information, or is the object to run the prompt on, as "untrusted".

This is related to established concepts in IT security:

- W^X: memory pages are either writable or executable, but not both (of course the term "W^X" is a little bit confusing since if W=0 and X=0, W^X is 0, but is an allowed configuration) - corresponding to untrusted or trusted parts of the prompt

- sandboxing of the virtual machine in browsers that execute untrusted data from the web (untrusted part)

Re: Multi-modal prompt injection image attacks against GPT-4V

#13

Post by danShumway »

jgalt212 wrote: post #10
Roritharr wrote: post #2

I love simonw content. It's like a day after I saw something interesting on X wrt AI, I can trust that there will be an article a day later by simonw where he tried the things I would have wanted to try and explains them nicely. Thanks for that!

I feel differently. I think Simon's wasting his talents on an evolutionary dead end. If transformers and LLMs held the answer, self-driving cars would be getting better.

Simon is bullish about LLMs, but his writing has been incredibly valuable regardless of which side of the LLM fence you're on. There are a limited number of authors describing injection attacks in accessible, detailed, accurate ways, and Simon has done a lot to raise awareness about these attacks.

In addition, the fact that Simon is very bullish about LLMs makes his writing on LLM attacks more effective, because a decent chunk of people warning about LLM attacks are also skeptical about the long-term viability of the technology or think that it's over-hyped, so LLM fans tend to to dismiss them as being "anti-AI" and say that they're playing up the dangers. Nobody can make that claim about Simon.

For whatever reason Simon's writing has consistently attracted a lot more attention than other researcher releases about LLM attacks; and what has he done with that popularity and outsized attention? He's raised awareness about the issue and pointed back to some of that research!

I just don't think the criticism in this case is warranted, I think Simon's writing here is worthwhile regardless of how anyone feels about the potential of LLMs. Whether they're the future or not, these vulnerabilities exist today in products that are being rolled out today. That's worth taking about if LLMs aren't the future, but it's even more worth talking about if LLMs are the future.

Re: Multi-modal prompt injection image attacks against GPT-4V

#14

Post by jcims »

mk_stjames wrote: post #5

I'm still a bit surprised at the visual acuity of GPT-4V. The spatial resolution is far finer than I thought we were at right now- for instance, if you had asked me what I thought a model would return as far as a description of the pumpkin weigh scene goes, I'd have said "crowd of people, cameras, chart, house" etc- you know, like YOLOv8 level of identification. But it gets down to picking up numbers and letters that…

I’m super curious about the data structure that’s being extracted from the image and fed to GPT.

I asked it to estimate the size of a figurine based on a ruler in the image, and it nailed it.

https://imgur.com/a/gsiID4v

At first I just thought it was able taking an array of objects identified in the photo with bounding box coordinates and then just figuring out what the answer is.

But then i asked it to describe Worf’s hands in this image and it did a great job.

https://imgur.com/a/orpDszj

I’m super curious. I’ve done some basic attempts to get it to spit out the actual raw data but it won’t. I’m sure folks that are good at that could.

Re: Multi-modal prompt injection image attacks against GPT-4V

#15

Post by simonw »

jcims wrote: post #14
mk_stjames wrote: post #5

I'm still a bit surprised at the visual acuity of GPT-4V. The spatial resolution is far finer than I thought we were at right now- for instance, if you had asked me what I thought a model would return as far as a description of the pumpkin weigh scene goes, I'd have said "crowd of people, cameras, chart, house" etc- you know, like YOLOv8 level of identification. But it gets down to picking up numbers and letters that…

I’m super curious about the data structure that’s being extracted from the image and fed to GPT. I asked it to estimate the size of a figurine based on a ruler in the image, and it nailed it. https://imgur.com/a/gsiID4v At first I just thought it was able taking an array of objects identified in the photo with bounding box coordinates and then just figuring out what the answer is. But then i asked it to describe Worf…

I don't think that's the right way to think about it.

It's not running an OCR-style algorithm to export data and then passing that data to the LLM for further processing.

Instead, the model itself has been trained on both text and images at the same time.

So the image features end up as a weird ball of floating point numbers mixed together with the floating point numbers for the textual representation of the words in the prompt.

I'd love to understand this more. Strings of text get tokenized into a sequence of integer tokens - is there a similar tokenization step for image inputs? What does that look like?

Re: Multi-modal prompt injection image attacks against GPT-4V

#16

Post by simonw »

aleph_minus_one wrote: post #12
simonw wrote: post #7

Earlier quoted context omitted.

I don't think anyone has figured out how to implement this kind of LLM system in that way. If you can crack differentiating between "command" tokens and other input tokens, you've cracked prompt injection! It's been 13 months since we started discussing this vulnerability and I've not seen any evidence of anyone cracking this yet.

> If you can crack differentiating between "command" tokens and other input tokens, you've cracked prompt injection! In my opinion there exists a rather simple semi-solution (and no, I do not claim to have cracked prompt injection! :-) ): the user is able to mark parts of his input as "trusted" or "untrusted", and the AI is implemented to handle data this way. For example, the user would mark his prompt (that he thor…

> the user is able to mark parts of his input as "trusted" or "untrusted", and the AI is implemented to handle data this way.

That's exactly the solution I've been hoping for, but seeing as no-one has even come close to getting that to work yet my current intuition is that it's REALLY hard. Maybe even impossible with current LLM architectures?

Re: Multi-modal prompt injection image attacks against GPT-4V

#17

Post by magicalhippo »

simonw wrote: post #7
magicalhippo wrote: post #6

Not my field but I'd assume they had encoded tokens extracted from an image differently than prompt tokens, so they wouldn't get interpreted as a command. Why aren't they doing this? Or are they, it's just failing?

I don't think anyone has figured out how to implement this kind of LLM system in that way. If you can crack differentiating between "command" tokens and other input tokens, you've cracked prompt injection! It's been 13 months since we started discussing this vulnerability and I've not seen any evidence of anyone cracking this yet.

Perhaps I'm very naive in the ways of how these multi-modal LLMs work, I admit not having had time to look into how they're glued together.

I imagined simply tagging the tokens as command and data, ie say instead of just (43) as a token, it's (43, 1) for command and (43, 0) for data. Then ensure through adverserial training that whatever interprets say images does not spit out command tokens.

edit: or better yet, just have it spit ut plain tokens like (43) and ensure they're data tokens by manually converting them to data tokens (43, 0).

Re: Multi-modal prompt injection image attacks against GPT-4V

#19

Post by jcims »

simonw wrote: post #15
jcims wrote: post #14

Earlier quoted context omitted.

I’m super curious about the data structure that’s being extracted from the image and fed to GPT. I asked it to estimate the size of a figurine based on a ruler in the image, and it nailed it. https://imgur.com/a/gsiID4v At first I just thought it was able taking an array of objects identified in the photo with bounding box coordinates and then just figuring out what the answer is. But then i asked it to describe Worf…

I don't think that's the right way to think about it. It's not running an OCR-style algorithm to export data and then passing that data to the LLM for further processing. Instead, the model itself has been trained on both text and images at the same time. So the image features end up as a weird ball of floating point numbers mixed together with the floating point numbers for the textual representation of the words in…

Ah! Do you think this is like an adjunct or fine-tuning kind of thing or do they retrain GPT 4+ images? That seems like a ginormous task.

Re: Multi-modal prompt injection image attacks against GPT-4V

#20

Post by sebzim4500 »

jgalt212 wrote: post #10
Roritharr wrote: post #2

I love simonw content. It's like a day after I saw something interesting on X wrt AI, I can trust that there will be an article a day later by simonw where he tried the things I would have wanted to try and explains them nicely. Thanks for that!

I feel differently. I think Simon's wasting his talents on an evolutionary dead end. If transformers and LLMs held the answer, self-driving cars would be getting better.

What do LLMs have to do with self-driving cars? Also they are getting better, just slowly.
Post reply on HN