Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Multi-modal prompt injection image attacks against GPT-4V

simonwillison.net

Post reply on HN
Print view View original

61–69 of 69 posts

Re: Multi-modal prompt injection image attacks against GPT-4V

#61

Post by simonw »

wunderwuzzi23 wrote: post #59
simonw wrote: post #11

Earlier quoted context omitted.

As a daily user of LLMs for over a year, I'm confident that they're not a waste of my time. Even if development were to freeze, they didn't improve at all from this day onwards, and their many monumental flaws (prompt injection, hallucination, inability to reason etc) were never solved, I still think they'd be worth studying and using. I expect we could still spend years figuring out new capabilities in the models we…

I'm very glad that you are focusing and helping educate the industry (both from pure tech but also security aspects). And agreed that as a daily user of LLMs myself, the potential (and practical use cases already) are enormous. E.g. Writing code and rapid prototyping will never be the same for me. By the way the last official answer I got from OpenAI regarding the markdown issue was that they thought it wouldn't be a…

Just added your tweet there to my Twitter thread about this: https://twitter.com/simonw/status/1713344669181178088

Re: Multi-modal prompt injection image attacks against GPT-4V

#62

Post by ewngzen »

jug wrote: post #56
jgalt212 wrote: post #10

Earlier quoted context omitted.

I feel differently. I think Simon's wasting his talents on an evolutionary dead end. If transformers and LLMs held the answer, self-driving cars would be getting better.

Well, self-driving cars are already in production and use. They may not be amazing just yet but they have already passed the greatest hurdle there is.

self-driving cars already in production are not explainable, at least with llm in it, we know what the network is thinking, and thus are possible to figure out what went wrong

Re: Multi-modal prompt injection image attacks against GPT-4V

#63

Post by jgalt212 »

sebzim4500 wrote: post #20
jgalt212 wrote: post #10

Earlier quoted context omitted.

I feel differently. I think Simon's wasting his talents on an evolutionary dead end. If transformers and LLMs held the answer, self-driving cars would be getting better.

What do LLMs have to do with self-driving cars? Also they are getting better, just slowly.

My analogy is that you cannot trust the outputs of an LLM or self-driving car.

Re: Multi-modal prompt injection image attacks against GPT-4V

#64

Post by Terretta »

danShumway wrote: post #9

> I was really surprised to see this work: I had assumed OpenAI would block Markdown images like this from loading content from external servers. Everyone always is because why on earth wouldn't this be blocked? That wuzzie has contacted OpenAI about this multiple times and gotten companies like Microsoft to say, "you know what, 3rd-party images are things that nobody allows in most contexts and it shouldn't be enabl…

The leak is not the remote image. The leak is the URL request.

Re: Multi-modal prompt injection image attacks against GPT-4V

#65

Post by Terretta »

gitfan86 wrote: post #18

This is the biggest problem in cyber security. You can tell someone to NEVER give out their TFA code over the phone and they will still do it when prompted by a phone call from their boss.

I'm going into a meeting and need to give some iTunes Gift Cards as an award. Can you grab me a few from an Apple store?

-- from your new boss, according to LinkedIn data

Re: Multi-modal prompt injection image attacks against GPT-4V

#66

Post by danShumway »

Terretta wrote: post #64
danShumway wrote: post #9

> I was really surprised to see this work: I had assumed OpenAI would block Markdown images like this from loading content from external servers. Everyone always is because why on earth wouldn't this be blocked? That wuzzie has contacted OpenAI about this multiple times and gotten companies like Microsoft to say, "you know what, 3rd-party images are things that nobody allows in most contexts and it shouldn't be enabl…

The leak is not the remote image. The leak is the URL request.

The remote image is the URL request, it gets made when the remote image is fetched by the front-end to render within the LLM's markdown response.

Re: Multi-modal prompt injection image attacks against GPT-4V

#67

Post by potatoman22 »

mk_stjames wrote: post #48
simonw wrote: post #15

Earlier quoted context omitted.

I don't think that's the right way to think about it. It's not running an OCR-style algorithm to export data and then passing that data to the LLM for further processing. Instead, the model itself has been trained on both text and images at the same time. So the image features end up as a weird ball of floating point numbers mixed together with the floating point numbers for the textual representation of the words in…

The entire GPT4 model was not retrained to utilize images, instead I think it is much more like show in [1] where an extra layer(s) are learned that take another pretrained image model and map that model to the latent space of the LLM. In [1] this was just a single linear layer. I found it pretty shocking how simple it was. I believe OpenAI is doing something similar. [1] https://arxiv.org/pdf/2304.10592.pdf

GPT4 was trained to be multimodal from the start.

Re: Multi-modal prompt injection image attacks against GPT-4V

#68

Post by Terretta »

danShumway wrote: post #66
Terretta wrote: post #64

Earlier quoted context omitted.

The leak is not the remote image. The leak is the URL request.

The remote image is the URL request, it gets made when the remote image is fetched by the front-end to render within the LLM's markdown response.

Yes. The remote image is not the leak. The URL request is the leak.

If this is confusing, control the server and have it return an image, or 404 instead of an image, doesn't matter, while logging the request.

You will find the exfiltrated data in the request log, and you will not find an image in the request log.

Of course, if your request handler is an image generator, you can generate an image containing your URL request. But by the time your generator gets the request, the data was leaked.

Re: Multi-modal prompt injection image attacks against GPT-4V

#69

Post by danShumway »

Terretta wrote: post #68
danShumway wrote: post #66

Earlier quoted context omitted.

The remote image is the URL request, it gets made when the remote image is fetched by the front-end to render within the LLM's markdown response.

Yes. The remote image is not the leak. The URL request is the leak. If this is confusing, control the server and have it return an image, or 404 instead of an image, doesn't matter, while logging the request. You will find the exfiltrated data in the request log, and you will not find an image in the request log. Of course, if your request handler is an image generator, you can generate an image containing your URL r…

The URL request is the browser request for the URL placed in the src tag of the image. The exfiltration technique is an image request. What the server returns as a response to that request doesn't matter. It doesn't even need to return anything at all.

If remote images were blocked from markdown as they are in many other applications, the URL request would not be made by the browser. The mitigation is that ChatGPT should refuse to render image tags or make requests for images if given a 3rd-party URL. I'm not sure if we're talking past each other or what, but I'm not talking about whether the server returns an image. I'm saying, ChatGPT will request an image from an external domain as part of rendering markdown during a conversation, and that GET request can be used as an exfiltration method.

The response from the server has nothing to do with it. I'm talking about the fact that ChatGPT will request arbitrary URLs without user input if the URLs are formatted in a markdown block as an image src.

ChatGPT is rendering this:
And it should instead when encountering a URL for a 3rd-party domain render this:
Post reply on HN