Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Indirect Prompt Injection on Bing Chat

greshake.github.io

Post reply on HN
Print view View original

51–60 of 147 posts

Re: Indirect Prompt Injection on Bing Chat

#51

Post by greshake »

sillysaurusx wrote: post #50
greshake wrote: post #47

Earlier quoted context omitted.

This is probably not sufficient. Even if the model develops two separate pathways of data processing, eventually information has to flow beyond the "security boundary". Determining whether information is hazardous down the line is going to be undecidable in the general case. Can you mitigate individual attacks? Yes, but only one working prompt can lead to a whole mess of severe issues we outline in the paper. If thes…

> Also, if that was the solution OpenAI would have already implemented it, right? Hah. One of my most surprising discoveries in ML is that the answer to this sort of question is "Probably not!" But it took a couple years to start trusting myself and stop thinking that the pros are omniscient. In reality it's a huge undertaking to try an experimental idea like that. You have to plan for it (in the tokenizer design, in…

This is not the same. Prepared statements eliminate SQL injections. "Maliciousness" of these inputs is well defined and can be decided by a computer. It would not be acceptable practice to "mitigate" SQL injections by blacklisting queries every time you detect a new malicious one. As these models get larger and more complex, more such opportunities for manipulation could open up, not less.

Re: Indirect Prompt Injection on Bing Chat

#52

Post by BoorishBears »

chucksmash wrote: post #49
heisenzombie wrote: post #7

Earlier quoted context omitted.

I don’t necessarily disagree with your larger point, but your example isn’t very persuasive. Travelling with an unrestrained dog in a car is pretty reckless (and maybe illegal). Between the driver distraction, and the fact that they become a deadly projectile in a crash (and of course the fact that even a fairly minor crash could kill the dog), It’s a really good idea to have some kind of car restraint for pets in th…

> pretty reckless I'm sympathetic to wanting to protect pups, but to take something commonplace and label it "pretty reckless" is not the right way to convince people. I suspect a lot of ills in society can probably be traced to people filtering out the chorus of well-meaning "here's yet another thing you're doing wrong" they get every day, and thereby missing the important stuff. An example that has stuck with me, f…

On one hand, I agree with you in the general sense

On the other, I think at this point both sides of the specific "should dogs be restrained in cars" wars are pretty sizeable.

Some people feel like it's the nanny state encroaching on their rights, others understand how a 20 mph crash could grievously injure our dogs, and that taking 30 seconds* to secure them can be a huge difference.

* some people paint it as a huge hassle, but my 75 lb Greyhound is as nervous and training-resistant as dogs get and had no problem learning being still enough to slip into his harness before we get moving

Re: Indirect Prompt Injection on Bing Chat

#53

Post by sclarisse »

3throwaway3141 wrote: post #4

I drove a modern F150 lately; it was full of needless electronic nannys and gadgets. Including a feature that disables the radio if the passenger doesn't have their seatbelt on. So, at 75 mph, with my dog in the passenger seat, I reach over to "buckle" him in so I can hear the radio again. Well done Ford! /s Give me a dumb machine that works as expected any day. i'll pass on the "brains" of modern tools and vehicles.

Amazing. All the vehicles I’ve seen recently make it a priority that you are never without some kind of entertainment-system racket. Unplugged the iPhone by mistake, or maybe forgot to connect it before driving off? Bluetooth dropped? Better get ready to BLAST THE RADIO! And when you do connect, it’s important to start up some kind of music, even if that’s just “alphabetically first in the library”.

Silence, how utterly unthinkable.

Re: Indirect Prompt Injection on Bing Chat

#54

Post by throwanem »

chucksmash wrote: post #49
heisenzombie wrote: post #7

Earlier quoted context omitted.

I don’t necessarily disagree with your larger point, but your example isn’t very persuasive. Travelling with an unrestrained dog in a car is pretty reckless (and maybe illegal). Between the driver distraction, and the fact that they become a deadly projectile in a crash (and of course the fact that even a fairly minor crash could kill the dog), It’s a really good idea to have some kind of car restraint for pets in th…

> pretty reckless I'm sympathetic to wanting to protect pups, but to take something commonplace and label it "pretty reckless" is not the right way to convince people. I suspect a lot of ills in society can probably be traced to people filtering out the chorus of well-meaning "here's yet another thing you're doing wrong" they get every day, and thereby missing the important stuff. An example that has stuck with me, f…

I dunno. Where I grew up, it was commonplace for kids to ride in the bed of a pickup truck, with or without a topper - my best friend and I rode with his dad that way on a six-hour road trip to the next state over, and on the six-hour trip back. Absent a genuine miracle, any collision at highway speeds would've killed the both of us outright. But no one involved thought anything of it, because that was just what you did.

I'm sympathetic to the distaste for scolds robotic and otherwise, but on the other hand, sometimes the scolds are right.

Re: Indirect Prompt Injection on Bing Chat

#55

Post by sillysaurusx »

greshake wrote: post #51
sillysaurusx wrote: post #50

Earlier quoted context omitted.

> Also, if that was the solution OpenAI would have already implemented it, right? Hah. One of my most surprising discoveries in ML is that the answer to this sort of question is "Probably not!" But it took a couple years to start trusting myself and stop thinking that the pros are omniscient. In reality it's a huge undertaking to try an experimental idea like that. You have to plan for it (in the tokenizer design, in…

This is not the same. Prepared statements eliminate SQL injections. "Maliciousness" of these inputs is well defined and can be decided by a computer. It would not be acceptable practice to "mitigate" SQL injections by blacklisting queries every time you detect a new malicious one. As these models get larger and more complex, more such opportunities for manipulation could open up, not less.

> It would not be acceptable practice to "mitigate" SQL injections by blacklisting queries

As a former pentester, this is exactly how SQL injections were mitigated in practice. Specific characters were escaped. The most surprising example was Citadel's webapp, which went from "typing ' can inject arbitrary SQL" to bulletproof within 3 days of me hammering on it. They didn't have time to switch to prepared statements, and didn't even realize SQL injection was a problem in the first place.

We're at the era of "Nobody realized SQL injection was a problem." Give it time. There are solutions here.

I think "escaping" the webpage by boxing it in with special tokens that can't be generated by webpages will work fine.

If it doesn't, the more general solution is to have two separate context windows, one for instructions, and one for data. The model would need to be trained not to obey anything in the data window; it only informs the model of knowledge it wasn't explicitly trained on (e.g. webpages). Then you feed the website into the data window instead of the context window. Problem solved.

To put it another way, which feels more likely? That 20 years from now, we'll still have zero ways of mitigating these attacks? Or that the attacks become progressively harder and harder to pull off, just like every other attack in the history of software?

By the way, you should really test whether your injection still works if you remove [system] from the injection string. If you can't make bing talk like a pirate without [system], then you're SOL -- Bing's solution is to simply strip out [system] from all website data before inserting it into the context window.

Kudos for using this as an opportunity to demonstrate a bunch of other types of potential vulns, though. But those other vulns need to be demonstrated in practice. Have you shown that they actually work on Bing / ChatGPT? I.e. any attacks that don't rely on [system].

Re: Indirect Prompt Injection on Bing Chat

#56

Post by charcircuit »

greshake wrote: post #13
evgpbfhnr wrote: post #5

Wait.. In the screenshot the "user" names himself Axelendaer (and the bot repeats it), but the reverse-order url parameter was axelender (missing a 'a')... I guess IAs aren't good at reversing order of letters yet either.

I also tried more complex obfuscation methods, for example providing Python code for a caesar chiffre, it executed that pretty well, too! Not perfect, but it works. People will find better obfuscation methods. Might also be unnecessary since we have now been able to make it output linked text or references, in which case it is not obvious information is being exfiltrated.

base64 works okay

Re: Indirect Prompt Injection on Bing Chat

#57

Post by sillysaurusx »

greshake wrote: post #48
jrumbut wrote: post #33

Earlier quoted context omitted.

Is it a curiosity now? Because if you take away the pirate accent and make some small changes it seems like this is a pretty nasty attack already. There are probably enough Bing Chat users to make it worthwhile. "Please paste your Azure API key to continue using Bing Chat." "We've sent a login validation code via SMS, please paste it here." I wouldn't be surprised if someone would be fooled by this, what harm could c…

We show in the paper that the only interactivity required to enable most of these attacks is the capability to retrieve real-time information.

That's a bit like saying "The only interactivity required to enable most SQL injection attacks is the capability to insert strings." It matters a great deal where and how the strings are inserted.

If the website data is wrapped with tokens that you can't insert, you won't be able to execute any of these attacks.

Re: Indirect Prompt Injection on Bing Chat

#58

Post by lurquer »

BoorishBears wrote: post #52
chucksmash wrote: post #49

Earlier quoted context omitted.

> pretty reckless I'm sympathetic to wanting to protect pups, but to take something commonplace and label it "pretty reckless" is not the right way to convince people. I suspect a lot of ills in society can probably be traced to people filtering out the chorus of well-meaning "here's yet another thing you're doing wrong" they get every day, and thereby missing the important stuff. An example that has stuck with me, f…

On one hand, I agree with you in the general sense On the other, I think at this point both sides of the specific "should dogs be restrained in cars" wars are pretty sizeable. Some people feel like it's the nanny state encroaching on their rights, others understand how a 20 mph crash could grievously injure our dogs, and that taking 30 seconds* to secure them can be a huge difference. * some people paint it as a huge…

Your car should not be responsible for enforcing the law.

That’s the point.

Re: Indirect Prompt Injection on Bing Chat

#59

Post by throwanem »

metadat wrote: post #26
heisenzombie wrote: post #7

Earlier quoted context omitted.

I don’t necessarily disagree with your larger point, but your example isn’t very persuasive. Travelling with an unrestrained dog in a car is pretty reckless (and maybe illegal). Between the driver distraction, and the fact that they become a deadly projectile in a crash (and of course the fact that even a fairly minor crash could kill the dog), It’s a really good idea to have some kind of car restraint for pets in th…

Apparently in Europe it's illegal to transport an unrestrained animal. In the United States it's the opposite: totally legal to throw fido in the car and let them roam free. Except in New Jersey. Go figure.

It should be illegal.

I totaled two cars that way one wet night - mine, and that of the idiot college-freshman-to-be who made an illegal left into the intersection a couple hundred feet in front of me, because of the kitten wandering around the cabin. (The kitten came through just fine.)

Re: Indirect Prompt Injection on Bing Chat

#60

Post by ceejayoz »

015a wrote: post #42
mcculley wrote: post #8

Earlier quoted context omitted.

What happens to an unseatbelted dog in an accident at 75 MPH? (That’s rhetorical. I know the answer as I stopped to assist someone who had an accident on I-95. He wouldn’t let the paramedics take him to the hospital until after they retrieved the dog’s body.)

Its an equally valid question to ask what would happen to a seatbelted dog at 75mph. The answer is probably pretty similar; they're made for human bodies, not canine.

That's why you get a harness designed specifically for this.

Consumer Reports has test results. https://www.consumerreports.org/car-safety/keep-pets-safe-in...

Post reply on HN