Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Multi-modal prompt injection image attacks against GPT-4V

simonwillison.net

Post reply on HN
Print view View original

51–60 of 69 posts

Re: Multi-modal prompt injection image attacks against GPT-4V

#51

Post by [deleted] »

simonw wrote: post #36
PeterisP wrote: post #23

Earlier quoted context omitted.

There are two main issues in my mind. First is that the straightforward way to implement any changes to the data structure requires full retraining of the model from scratch, which is very expensive for the large models (I recall reading that GPT-4 training compute cost was sixty million or so) - so it's not something someone would or could do 'for fun' just to test a hypothesis, there needs to be a will to explicitl…

Anyone who can build an API to an LLM that was provably secure against prompt injection would have a license to print money right now. I don't think the lack of a solution is because it's too expensive. It's because we don't know how to do it yet.

[deleted]

Re: Multi-modal prompt injection image attacks against GPT-4V

#52

Post by magicalhippo »

simonw wrote: post #45
magicalhippo wrote: post #44

Earlier quoted context omitted.

Do you have some references (articles, blogs or similar) that illuminates why the trivial solutions won't work? Clearly I know enough to come up with a trivial solution, but not enough to immediately see why it won't work.

I've written a lot about this myself: https://simonwillison.net/series/prompt-injection/

Excellent, thank you.

Re: Multi-modal prompt injection image attacks against GPT-4V

#53

Post by PeterisP »

simonw wrote: post #36
PeterisP wrote: post #23

Earlier quoted context omitted.

There are two main issues in my mind. First is that the straightforward way to implement any changes to the data structure requires full retraining of the model from scratch, which is very expensive for the large models (I recall reading that GPT-4 training compute cost was sixty million or so) - so it's not something someone would or could do 'for fun' just to test a hypothesis, there needs to be a will to explicitl…

Anyone who can build an API to an LLM that was provably secure against prompt injection would have a license to print money right now. I don't think the lack of a solution is because it's too expensive. It's because we don't know how to do it yet.

Despite paying it a lot of lip service, in practice companies demonstrate that they don't really care about security that much, so I don't think that resistance to prompt injection is the main blocker for any products right now; if some product was very useful but had issues with prompt injection, then people would complain a lot about it while still using it and paying for it; and security (much less the even more niche requirement of provable security) would only be a differentiator among multiple competitors providing a comparable product, not a barrier for selling the product in the first place.

Re: Multi-modal prompt injection image attacks against GPT-4V

#54

Post by simonw »

PeterisP wrote: post #53
simonw wrote: post #36

Earlier quoted context omitted.

Anyone who can build an API to an LLM that was provably secure against prompt injection would have a license to print money right now. I don't think the lack of a solution is because it's too expensive. It's because we don't know how to do it yet.

Despite paying it a lot of lip service, in practice companies demonstrate that they don't really care about security that much, so I don't think that resistance to prompt injection is the main blocker for any products right now; if some product was very useful but had issues with prompt injection, then people would complain a lot about it while still using it and paying for it; and security (much less the even more n…

I think prompt injection is the main reason there's been buzz about the idea of autonomous agents and digital personal assistants for months but no one has released a convincing product yet.

Here's an example of an agents company that pivoted partly because they couldn't figure out how to secure them: https://twitter.com/zachtratar/status/1694024240880861571

Re: Multi-modal prompt injection image attacks against GPT-4V

#55

Post by jug »

simonw wrote: post #15
jcims wrote: post #14

Earlier quoted context omitted.

I’m super curious about the data structure that’s being extracted from the image and fed to GPT. I asked it to estimate the size of a figurine based on a ruler in the image, and it nailed it. https://imgur.com/a/gsiID4v At first I just thought it was able taking an array of objects identified in the photo with bounding box coordinates and then just figuring out what the answer is. But then i asked it to describe Worf…

I don't think that's the right way to think about it. It's not running an OCR-style algorithm to export data and then passing that data to the LLM for further processing. Instead, the model itself has been trained on both text and images at the same time. So the image features end up as a weird ball of floating point numbers mixed together with the floating point numbers for the textual representation of the words in…

I wonder if OpenAI themselves even know how the resulting black box actually operates? They know how to train it, sure, but do they understand how it approaches the end results?

Re: Multi-modal prompt injection image attacks against GPT-4V

#56

Post by jug »

jgalt212 wrote: post #10
Roritharr wrote: post #2

I love simonw content. It's like a day after I saw something interesting on X wrt AI, I can trust that there will be an article a day later by simonw where he tried the things I would have wanted to try and explains them nicely. Thanks for that!

I feel differently. I think Simon's wasting his talents on an evolutionary dead end. If transformers and LLMs held the answer, self-driving cars would be getting better.

Well, self-driving cars are already in production and use. They may not be amazing just yet but they have already passed the greatest hurdle there is.

Re: Multi-modal prompt injection image attacks against GPT-4V

#57

Post by wunderwuzzi23 »

danShumway wrote: post #9

> I was really surprised to see this work: I had assumed OpenAI would block Markdown images like this from loading content from external servers. Everyone always is because why on earth wouldn't this be blocked? That wuzzie has contacted OpenAI about this multiple times and gotten companies like Microsoft to say, "you know what, 3rd-party images are things that nobody allows in most contexts and it shouldn't be enabl…

Thanks for the shout out, the last answer I got from OpenAI in that regards was that they thought it wouldn't be a problem because there will be mitigations. My assumption now is that they thought they could maybe fix indirect prompt injection, but it doesn't look like there is a solution.

My conclusion is that with LLMs, the assumption always needs to be that the LLM might be capable of outputting any content, and the client needs to safely handle whatever is returned.

Re: Multi-modal prompt injection image attacks against GPT-4V

#58

Post by simonw »

jug wrote: post #55
simonw wrote: post #15

Earlier quoted context omitted.

I don't think that's the right way to think about it. It's not running an OCR-style algorithm to export data and then passing that data to the LLM for further processing. Instead, the model itself has been trained on both text and images at the same time. So the image features end up as a weird ball of floating point numbers mixed together with the floating point numbers for the textual representation of the words in…

I wonder if OpenAI themselves even know how the resulting black box actually operates? They know how to train it, sure, but do they understand how it approaches the end results?

I'm pretty confident they don't, and I think they might openly admit that.

There's a lot of ongoing research into interpretability of these models. Anthropic put out a major new paper about that a few days ago: https://www.anthropic.com/index/decomposing-language-models-...

Re: Multi-modal prompt injection image attacks against GPT-4V

#59

Post by wunderwuzzi23 »

simonw wrote: post #11
jgalt212 wrote: post #10

Earlier quoted context omitted.

I feel differently. I think Simon's wasting his talents on an evolutionary dead end. If transformers and LLMs held the answer, self-driving cars would be getting better.

As a daily user of LLMs for over a year, I'm confident that they're not a waste of my time. Even if development were to freeze, they didn't improve at all from this day onwards, and their many monumental flaws (prompt injection, hallucination, inability to reason etc) were never solved, I still think they'd be worth studying and using. I expect we could still spend years figuring out new capabilities in the models we…

I'm very glad that you are focusing and helping educate the industry (both from pure tech but also security aspects). And agreed that as a daily user of LLMs myself, the potential (and practical use cases already) are enormous. E.g. Writing code and rapid prototyping will never be the same for me.

By the way the last official answer I got from OpenAI regarding the markdown issue was that they thought it wouldn't be a problem because there will be mitigations. My assumption now is that they thought they could maybe fix indirect prompt injection, but it doesn't look like there is a solution.

My current takeaway is that with LLMs, the assumption always needs to be that the LLM is capable of outputting any content, and the client needs to safely (and securely) handle whatever is returned.

Btw. I show cased these kind of image based prompt injections in July (with Bard and Bing Chat, which presumably already used OpenAI's tech), and the example I created (the robot slipping on a banana) was shown during Blackhat in Las Vegas this year. Tweets: https://twitter.com/wunderwuzzi23/status/1681520761146834946

Re: Multi-modal prompt injection image attacks against GPT-4V

#60

Post by danShumway »

dragonwriter wrote: post #47
simonw wrote: post #39

Earlier quoted context omitted.

Because it's true, right? I'm not saying there aren't any HARD steps. Every time I talk about prompt injection the same "surely there's an easy fix: ...." comments show up. I'm just trying to shortcut having to argue against them all again.

> Because it's true, right? No, while sometimes there are barriers that take a giant leap to cross, progress tends to involveots of steps that were simple, even if time consuming to execute, once the preceding pieces were in place.

After over a year of hearing people say this, at some point the burden of proof shifts to the people making the claim.

There was a period where LLMs were extremely inaccessible to train where it would have been reasonable to say "the mitigation is easy, but OpenAI hasn't done it." But nowadays, indie models and freeware models are much more common, and there are even models with open datasets. Fine tuning can be done on a Macbook. Purchasing compute to fully replicate a smaller model is within reach of a lot of people.

There is still a cost involved, it's not trivial -- but it's a lot more accessible than it used to be. So at what point is it reasonable to respond to the people saying, "it's easy, just do X" with "well, do it then."?

If the training method you're describing works, then what is preventing someone from training a 7B model over the next week that's invulnerable to prompt injection?

Post reply on HN