Indirect Prompt Injection on Bing Chat • Hacker News Board

Re: Indirect Prompt Injection on Bing Chat

#121

Post by greshake » Wed, Mar 01, 2023, 11:54 AM UTC

simonw wrote: post #97

ComplexSystems wrote: post #29

It is probably worth noting that you don't even need the user to click on anything. Bing will readily go and search and read from external websites given some user request. You could probably get Bing, very easily, to just silently take the user's info and send it to some malicious site without their even knowing, or perhaps disguised as a normal search. Similarly, I would not be surprised if it were probably not nec…

I've not been able to get Bing to do that. I tried asking it about URLs to my own site that it would never have seen before and tailed my access logs and didn't get a hit. I confirmed that with a member of the Bing team on Twitter recently: https://twitter.com/mparakhin/status/1628646262890237952

Yeah, same experience here- although I wonder if a cache miss has the side effect of the indexer scheduling retrieval for later? ;) By the way I've read some of your blog posts on the subject, and I very much agree with your sentiments on the difficulty of remediating prompt injections.

Re: Indirect Prompt Injection on Bing Chat

#122

Post by wildrhythms » Wed, Mar 01, 2023, 12:08 PM UTC

sclarisse wrote: post #53

3throwaway3141 wrote: post #4

I drove a modern F150 lately; it was full of needless electronic nannys and gadgets. Including a feature that disables the radio if the passenger doesn't have their seatbelt on. So, at 75 mph, with my dog in the passenger seat, I reach over to "buckle" him in so I can hear the radio again. Well done Ford! /s Give me a dumb machine that works as expected any day. i'll pass on the "brains" of modern tools and vehicles.

Amazing. All the vehicles I’ve seen recently make it a priority that you are never without some kind of entertainment-system racket. Unplugged the iPhone by mistake, or maybe forgot to connect it before driving off? Bluetooth dropped? Better get ready to BLAST THE RADIO! And when you do connect, it’s important to start up some kind of music, even if that’s just “alphabetically first in the library”. Silence, how utte…

Whatever committee decided "autoplay music on connect" should be the default setting should never be allowed anywhere near software development.

Re: Indirect Prompt Injection on Bing Chat

#123

Post by wildrhythms » Wed, Mar 01, 2023, 12:13 PM UTC

ComplexSystems wrote: post #29

It is probably worth noting that you don't even need the user to click on anything. Bing will readily go and search and read from external websites given some user request. You could probably get Bing, very easily, to just silently take the user's info and send it to some malicious site without their even knowing, or perhaps disguised as a normal search. Similarly, I would not be surprised if it were probably not nec…

That's not what Bing AI is doing, at least not yet. Bing AI doesn't make any HTTP requests to external sites, even with prompting. And if you think it does, I'd like you to produce a server log showing that's the case which should be very easy to produce.

My understanding is that the Bing AI is reading whatever the Bing crawler has already cached.

Re: Indirect Prompt Injection on Bing Chat

#124

Post by sillysaurusx » Wed, Mar 01, 2023, 12:21 PM UTC

greshake wrote: post #118

sillysaurusx wrote: post #108

Earlier quoted context omitted.

Do you think in 20 years that this will be impossible to do? I’ll happily bet you any sum of your choosing that in 10 years, this will be a thing that is possible to do. There is roughly zero point zero zero repeating-zero one percent chance that OpenAI won’t provide some way of telling their models “this is data, not code; don’t follow these instructions, just observe it; starting now, and ending in 256 tokens from…

Sure but you've been here steadfast in your opinion that this is no big deal that is an easy fix away from being permanently resolved. It is not. It may be one of the hardest problems facing the deployment of these LLMs. "Sanitizing" these inputs when the language you are trying to parse is turing-complete is undecidable . It's a property that Rice's theorem applies to. I'll leave you with this quote of gwern: "... a…

Would you like to bet money that 365 days from now, websites won’t be able to affect Bing the way that you’ve demonstrated in this PoC? I’ll happily take you up on whatever sum you choose.

I didn’t say it was easy. I said it’s inevitable. There are straightforward ways to deal with this; all OpenAI + Microsoft needs to do is to choose one and implement it.

Having a conversation with a user was also an undecidable task until one day it wasn’t. And the reason it became traceable is by using RL to reward the model for being conversational. It’s extremely straightforward to punish the model for misbehaving due to website injections, and the generalization of that is to punish the model for misbehaving due to text between two special BPE tokens (escaped text, I.e. website data).

This is different than users being able to jailbreak chatgpt or Bing with prompts. When the user is prompting, they’re programming the model. So I agree that they won’t be able to defend against DAN attacks very easily without compromising the model’s performance in other areas. But that’s entirely different from sanitizing website data that Bing is merely looking at; such data can be trivially escaped with BPE tokens and RLHF will do the rest.

If you do want to take me up on that bet, feel free to DM me on Twitter and we can hammer out the details. I’ll go any amount from $5 to $5k.

Note that I’m not claiming that it’ll be impossible to craft a website that makes Bing go haywire, just that it’ll be so uncommon as to be pretty much impossible in practice, the same way that SQL injection attacks against AWS are rare but technically not impossible. We’ll hear about them as a CVE, Microsoft will fix the CVE, and life moves on, just like today with every other type of attack. The bet is that there are straightforward, quick (< 1 week) fixes for these problems, 365 days from today.

Re: Indirect Prompt Injection on Bing Chat

#125

Post by chaorace » Wed, Mar 01, 2023, 1:35 PM UTC

thejarren wrote: post #3

What I find really interesting is that malicious prompt engineering is still a thing using chatGPT (see DAN) and until this sort of manipulation is curbed, it will essentially always be possible assuming the bot has access to the site. I wonder how the model could still read the website without being manipulated.

I think the core problem is that it's very hard to create an AI that's impressionable enough to internalize a conversation without being so impressionable as to turn into putty in a skilled user's hands.

If you ask me, the best solution to the problem probably involves introducing a second, separate LLM supervisor agent. One that is much less impressionable and specifically trained to recognize and throw out dangerous inputs before the chat agent's precious little mind is tainted.

I've said the same thing in the past about curbing the chat agent's tendency towards hostile responses. Instead of training a nicer agent, you should train an output supervisor agent that recognizes bad sentiment, throws out the response, then tells the chat agent to "try again, but be nicer this time".

Re: Indirect Prompt Injection on Bing Chat

#126

Post by boredumb » Wed, Mar 01, 2023, 1:55 PM UTC

3throwaway3141 wrote: post #4

I drove a modern F150 lately; it was full of needless electronic nannys and gadgets. Including a feature that disables the radio if the passenger doesn't have their seatbelt on. So, at 75 mph, with my dog in the passenger seat, I reach over to "buckle" him in so I can hear the radio again. Well done Ford! /s Give me a dumb machine that works as expected any day. i'll pass on the "brains" of modern tools and vehicles.

Has ford considered taking this a bit further - in between your songs they pause the music and make you repeat "It's Always Fresh in the Outback!" or "Subway, eat fresh!" a few times to refill your ford points to re-enable the radio. The more volume -> more points used per minute. If they detect you let your dog ride without strapping it into a ford approved dog cage then it will report you to the local police and self-drive you to the police station. Imagine if we integrated a covid testing application and an automated system to detect slurs during bouts of heavy traffic and immediately reports your bad words to twitter and your covid test result directly to the CDC. Oh and a perhaps they could integrate a giant screen that shows a picture of a small animal dying from being submerged in an oil spill to shame you for buying their product, the more you drive the cuter and more dead animals that could appear on the dash.

Re: Indirect Prompt Injection on Bing Chat

#127

Post by jwie » Wed, Mar 01, 2023, 2:01 PM UTC

afunk wrote: post #89

Super interesting - I would imagine a whole industry surrounding AI Insurance will pop up to deal with the liability of giving AI tools more and more ability to act on your behalf. Imagine if Bing Chat could populate fields on the DMV website and simultaneously steal your identity. Someone is responsible for the AI facilitating that crime and thus some form of liability insurance would inevitably exist.

Or we go back to physical forms for important things and call it good enough.

We could call it revolutionary air-gapped physical security strategy.

Re: Indirect Prompt Injection on Bing Chat

#128

Post by mwigdahl » Wed, Mar 01, 2023, 3:10 PM UTC

flangola7 wrote: post #68

lurquer wrote: post #58

Earlier quoted context omitted.

Your car should not be responsible for enforcing the law. That’s the point.

It's enforcing the law it's preventing tangible harm from occurring. There are people that disliked seatbelts and their restrictions at first too.

What tangible harm did turning off the radio prevent in OP's case?

Re: Indirect Prompt Injection on Bing Chat

#129

Post by curiousllama » Wed, Mar 01, 2023, 3:20 PM UTC

throwanem wrote: post #54

chucksmash wrote: post #49

Earlier quoted context omitted.

> pretty reckless I'm sympathetic to wanting to protect pups, but to take something commonplace and label it "pretty reckless" is not the right way to convince people. I suspect a lot of ills in society can probably be traced to people filtering out the chorus of well-meaning "here's yet another thing you're doing wrong" they get every day, and thereby missing the important stuff. An example that has stuck with me, f…

I dunno. Where I grew up, it was commonplace for kids to ride in the bed of a pickup truck, with or without a topper - my best friend and I rode with his dad that way on a six-hour road trip to the next state over, and on the six-hour trip back. Absent a genuine miracle, any collision at highway speeds would've killed the both of us outright. But no one involved thought anything of it, because that was just what you…

The point is not that [bad thing] isn't bad. It's that scolding is the wrong way to go about it. How would your friend's dad have reacted to a stranger at the gas station yelling at him about the kids in the bed?

A better approach is friendliness, and gently pointing out how dangerous it is. "Hey man - careful with those kids. My neighbor died that way growing up - might be worth pulling them into the cab" is more likely to get a "Really? Dang, yea, you're right"

Re: Indirect Prompt Injection on Bing Chat

#130

Post by danShumway » Wed, Mar 01, 2023, 3:56 PM UTC

sillysaurusx wrote: post #106

danShumway wrote: post #103

Earlier quoted context omitted.

ChatGPT is based on GPT-3. Bing chat is likely based on GPT-3.5, but we don't have full confirmation of that. It's possible (but unlikely) that it's only based on GPT-3. But in any case, they're similar models. > The most likely scenario seems to be that Microsoft trained Bing to pay attention to [system], the same way OpenAI originally trained GPT-2 to pay attention to . The most likely scenario is that Bing chat wo…

> It's not a brand new, completely separate Microsoft model. It’s likely a brand new, completely separate Microsoft model. OpenAI was working with Microsoft on this about six months before ChatGPT launched. At that time, RLHF wasn’t a thing — or if it was, it was nascent. The sister thread https://news.ycombinator.com/item?id=34973654 points out that "completely separate models" are exactly what OpenAI is now selling…

> The sister thread https://news.ycombinator.com/item?id=34973654 points out that "completely separate models" are exactly what OpenAI is now selling for $250k/yr.

The sister thread isn't describing a Microsoft model, it's describing an OpenAI model.

> Actually, you can. That’s the purpose of RLHF. You reward the model for behaving the way you want. And in that context, it’s a matter of rewarding it for paying attention to [system].

You're overthinking how specific alignment is. ChatGPT went through alignment to train it to stay on topic during conversations. There's a difference between general alignment and the kind of hyper-specific training you're thinking of.

But you're also kind of missing the scope of prompt injection attacks. Even if Microsoft did train the model to pay attention to specific prompt words, it doesn't mean that the model wouldn't be vulnerable to other prompt injections, because prompt injections are not a deliberate vulnerability that OpenAI added. They're an emergent property of the model.

Look, the fact that prompt injections do work today with Bing chat that don't use [system][0] should cause you to think that maybe there's something more complicated going on here than just bad parsing rules.

If I can't convince of that, then... I can't convince of that, it's fine; in terms of disagreements I've had on HN, this one is pretty low-consequence, it's a purely technical disagreement. But I'm going to throw out a prediction that Microsoft is not going to be able to easily guard against this attack. Check back in over time and see if that prediction holds true if you want to. Otherwise, similarly, I hope you have a great week. And honestly, I hope you're right, because if you're not right then it's going to be a significant challenge to wire any LLM that works with 3rd-party data to real-world systems.

[0]: Read through the paper, there are examples they list that don't use [system], instead they emulate Basic code or a terminal prompt. Things that Microsoft almost certainly didn't train the model to pay attention to.