Multi-modal prompt injection image attacks against GPT-4V
simonwillison.net
Multi-modal prompt injection image attacks against GPT-4V
1–10 of 69 posts
Re: Multi-modal prompt injection image attacks against GPT-4V
#2I love simonw content. It's like a day after I saw something interesting on X wrt AI, I can trust that there will be an article a day later by simonw where he tried the things I would have wanted to try and explains them nicely. Thanks for that!
Re: Multi-modal prompt injection image attacks against GPT-4V
#3I love simonw content. It's like a day after I saw something interesting on X wrt AI, I can trust that there will be an article a day later by simonw where he tried the things I would have wanted to try and explains them nicely. Thanks for that!
Same sentiment. Its like he reads hacker news or similar sources and is generally on top of tech news and then has the curiosity to look more into it himself and do practical experiments. I try to do the same which is why I love this content.
Re: Multi-modal prompt injection image attacks against GPT-4V
#4Vision is going to be significantly harder to filter for due to the higher dimensional nature of the inputs.
Re: Multi-modal prompt injection image attacks against GPT-4V
#5I'm still a bit surprised at the visual acuity of GPT-4V. The spatial resolution is far finer than I thought we were at right now- for instance, if you had asked me what I thought a model would return as far as a description of the pumpkin weigh scene goes, I'd have said "crowd of people, cameras, chart, house" etc- you know, like YOLOv8 level of identification.
But it gets down to picking up numbers and letters that are tiny, gets obscured text like half moon bay (is it using a combination of the scene letters and knowledge in its language model of a pumpkin contest in HMB and synthesizing that knowledge into recognizing the location!!!!???). It's damn detailed.
Re: Multi-modal prompt injection image attacks against GPT-4V
#6Not my field but I'd assume they had encoded tokens extracted from an image differently than prompt tokens, so they wouldn't get interpreted as a command.
Why aren't they doing this? Or are they, it's just failing?
Re: Multi-modal prompt injection image attacks against GPT-4V
#7Not my field but I'd assume they had encoded tokens extracted from an image differently than prompt tokens, so they wouldn't get interpreted as a command. Why aren't they doing this? Or are they, it's just failing?
I don't think anyone has figured out how to implement this kind of LLM system in that way.
If you can crack differentiating between "command" tokens and other input tokens, you've cracked prompt injection!
It's been 13 months since we started discussing this vulnerability and I've not seen any evidence of anyone cracking this yet.
Re: Multi-modal prompt injection image attacks against GPT-4V
#8I'm still a bit surprised at the visual acuity of GPT-4V. The spatial resolution is far finer than I thought we were at right now- for instance, if you had asked me what I thought a model would return as far as a description of the pumpkin weigh scene goes, I'd have said "crowd of people, cameras, chart, house" etc- you know, like YOLOv8 level of identification. But it gets down to picking up numbers and letters that…
Yeah, I think it's spectacular - GPT4-V is so much better then I expected it to be.
Re: Multi-modal prompt injection image attacks against GPT-4V
#9> I was really surprised to see this work: I had assumed OpenAI would block Markdown images like this from loading content from external servers.
Everyone always is because why on earth wouldn't this be blocked?
That wuzzie has contacted OpenAI about this multiple times and gotten companies like Microsoft to say, "you know what, 3rd-party images are things that nobody allows in most contexts and it shouldn't be enabled in places like Bing chat" and OpenAI in particular seems to have consistently decided that it's a core part of the chat capabilities for some reason -- it's on of the reasons why I say that OpenAI does not take security seriously.
It really should not be a debate whether or not remote images should be blocked from markdown. Your email client blocks remote images. Most forums either block remote images or proxy them. There is no reason at all for ChatGPT to have that the ability to display them.
Every company makes security mistakes, that's not a problem; it's when a company doubles down on security mistakes that their attitude about security becomes apparent.
Re: Multi-modal prompt injection image attacks against GPT-4V
#10I love simonw content. It's like a day after I saw something interesting on X wrt AI, I can trust that there will be an article a day later by simonw where he tried the things I would have wanted to try and explains them nicely. Thanks for that!
I feel differently. I think Simon's wasting his talents on an evolutionary dead end. If transformers and LLMs held the answer, self-driving cars would be getting better.