Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Multi-modal prompt injection image attacks against GPT-4V

simonwillison.net

Post reply on HN
Print view View original

31–40 of 69 posts

Re: Multi-modal prompt injection image attacks against GPT-4V

#31

Post by spdustin »

danShumway wrote: post #9

> I was really surprised to see this work: I had assumed OpenAI would block Markdown images like this from loading content from external servers. Everyone always is because why on earth wouldn't this be blocked? That wuzzie has contacted OpenAI about this multiple times and gotten companies like Microsoft to say, "you know what, 3rd-party images are things that nobody allows in most contexts and it shouldn't be enabl…

I use it all the time to embed graphviz diagrams by instructing it to url-encode them and include it in the query string to a rendering endpoint. It spooked me enough that I added proxy layer to verify that it’s not rendering what appears to be a chat or PII.

The world isn’t ready for LLMs unleashed like this. Any plugin could do the same thing, and exfiltrate any part of the chat context with very little instruction engineering to bypass what things guardrails exist on the plugins side.

Re: Multi-modal prompt injection image attacks against GPT-4V

#32

Post by simonw »

saberience wrote: post #30
Michelangelo11 wrote: post #24

Earlier quoted context omitted.

Interesting that you say that, because this lines up exactly with my mental models of LLM image processing and LLM functioning in general. One of the defining features of LLMs is that, unlike in regular programming, there's no separation between data and instructions -- they're one and the same -- and that's what we're seeing here. Also, LLMs tend to focus more heavily on the instructions they see last (because of li…

So the original release of ChatGPT would call a URL you entered in the chat? I don't recall it acting that way. In the linked post ChatGPT literally goes to a URL from an image... i.e. it calls something like curl with that URL. This definitely seems like something adjacent to an LLM.

ChatGPT didn't grow the ability to visit URLs until they introduced ChatGPT Browse - though it frequently hallucinated that it could: https://simonwillison.net/2023/Mar/10/chatgpt-internet-acces...

The exfiltration demo in the blog post doesn't show ChatGPT fetching a URL: it shows it outputting an image tag with an src= pointing at an external URL, which the user's browser then attempts to load.

Re: Multi-modal prompt injection image attacks against GPT-4V

#33

Post by simonw »

dragonwriter wrote: post #29
simonw wrote: post #7

Earlier quoted context omitted.

I don't think anyone has figured out how to implement this kind of LLM system in that way. If you can crack differentiating between "command" tokens and other input tokens, you've cracked prompt injection! It's been 13 months since we started discussing this vulnerability and I've not seen any evidence of anyone cracking this yet.

> If you can crack differentiating between "command" tokens and other input tokens, you've cracked prompt injection! Isn't this simple? Have two different tokenizer with non-overlapping output ranges, one for “trusted commands” and one for “user input”, do training with various mixes of commands and input with the appropriate reactions to each, and then have the trusted command fixed (or selectable from a small set o…

Definitely not simple. If it was simple we would have solved it already!

Re: Multi-modal prompt injection image attacks against GPT-4V

#34

Post by beebmam »

simonw wrote: post #8
mk_stjames wrote: post #5

I'm still a bit surprised at the visual acuity of GPT-4V. The spatial resolution is far finer than I thought we were at right now- for instance, if you had asked me what I thought a model would return as far as a description of the pumpkin weigh scene goes, I'd have said "crowd of people, cameras, chart, house" etc- you know, like YOLOv8 level of identification. But it gets down to picking up numbers and letters that…

Yeah, I think it's spectacular - GPT4-V is so much better then I expected it to be.

It's fun to ask it to try to draw ascii art based on an image. It barely draws anything resembling the original image, but usually gets the concepts in the original image accurately.

Re: Multi-modal prompt injection image attacks against GPT-4V

#35

Post by Michelangelo11 »

saberience wrote: post #30
Michelangelo11 wrote: post #24

Earlier quoted context omitted.

Interesting that you say that, because this lines up exactly with my mental models of LLM image processing and LLM functioning in general. One of the defining features of LLMs is that, unlike in regular programming, there's no separation between data and instructions -- they're one and the same -- and that's what we're seeing here. Also, LLMs tend to focus more heavily on the instructions they see last (because of li…

So the original release of ChatGPT would call a URL you entered in the chat? I don't recall it acting that way. In the linked post ChatGPT literally goes to a URL from an image... i.e. it calls something like curl with that URL. This definitely seems like something adjacent to an LLM.

Well, the original release had browsing disabled, so it couldn't access any URL.

Re: Multi-modal prompt injection image attacks against GPT-4V

#36

Post by simonw »

PeterisP wrote: post #23
simonw wrote: post #16

Earlier quoted context omitted.

> the user is able to mark parts of his input as "trusted" or "untrusted", and the AI is implemented to handle data this way. That's exactly the solution I've been hoping for, but seeing as no-one has even come close to getting that to work yet my current intuition is that it's REALLY hard. Maybe even impossible with current LLM architectures?

There are two main issues in my mind. First is that the straightforward way to implement any changes to the data structure requires full retraining of the model from scratch, which is very expensive for the large models (I recall reading that GPT-4 training compute cost was sixty million or so) - so it's not something someone would or could do 'for fun' just to test a hypothesis, there needs to be a will to explicitl…

Anyone who can build an API to an LLM that was provably secure against prompt injection would have a license to print money right now.

I don't think the lack of a solution is because it's too expensive. It's because we don't know how to do it yet.

Re: Multi-modal prompt injection image attacks against GPT-4V

#37

Post by Michelangelo11 »

aleph_minus_one wrote: post #12
simonw wrote: post #7

Earlier quoted context omitted.

I don't think anyone has figured out how to implement this kind of LLM system in that way. If you can crack differentiating between "command" tokens and other input tokens, you've cracked prompt injection! It's been 13 months since we started discussing this vulnerability and I've not seen any evidence of anyone cracking this yet.

> If you can crack differentiating between "command" tokens and other input tokens, you've cracked prompt injection! In my opinion there exists a rather simple semi-solution (and no, I do not claim to have cracked prompt injection! :-) ): the user is able to mark parts of his input as "trusted" or "untrusted", and the AI is implemented to handle data this way. For example, the user would mark his prompt (that he thor…

> and the AI is implemented to handle data this way.

This is the key issue. The problem is, I don't see how to reliably create that data handling implementation -- it always comes down to the fact that any feasible implementation takes the form of text that gets passed to the AI, and clever prompt hacking can override the instructions in that text.

Re: Multi-modal prompt injection image attacks against GPT-4V

#38

Post by dragonwriter »

simonw wrote: post #33
dragonwriter wrote: post #29

Earlier quoted context omitted.

> If you can crack differentiating between "command" tokens and other input tokens, you've cracked prompt injection! Isn't this simple? Have two different tokenizer with non-overlapping output ranges, one for “trusted commands” and one for “user input”, do training with various mixes of commands and input with the appropriate reactions to each, and then have the trusted command fixed (or selectable from a small set o…

Definitely not simple. If it was simple we would have solved it already!

Always love the “there are no easy next steps from whwre we currently are, because if they were someone would already have taken them” argument.

Re: Multi-modal prompt injection image attacks against GPT-4V

#39

Post by simonw »

dragonwriter wrote: post #38
simonw wrote: post #33

Earlier quoted context omitted.

Definitely not simple. If it was simple we would have solved it already!

Always love the “there are no easy next steps from whwre we currently are, because if they were someone would already have taken them” argument.

Because it's true, right?

I'm not saying there aren't any HARD steps.

Every time I talk about prompt injection the same "surely there's an easy fix: ...." comments show up. I'm just trying to shortcut having to argue against them all again.

Re: Multi-modal prompt injection image attacks against GPT-4V

#40

Post by nullc »

It sure seems that a lot of effort goes in to preventing the LLM from obeying the user while actual security issues-- like allowing CSRF like attacks from the interface-- go ignored.

If the LLM embedding content from arbitrary sources is really such a critical feature it could at least be behind a click that shows you the URL.

Post reply on HN