Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Indirect Prompt Injection on Bing Chat

greshake.github.io

Post reply on HN
Print view View original

61–70 of 147 posts

Re: Indirect Prompt Injection on Bing Chat

#61

Post by throwanem »

sclarisse wrote: post #53
3throwaway3141 wrote: post #4

I drove a modern F150 lately; it was full of needless electronic nannys and gadgets. Including a feature that disables the radio if the passenger doesn't have their seatbelt on. So, at 75 mph, with my dog in the passenger seat, I reach over to "buckle" him in so I can hear the radio again. Well done Ford! /s Give me a dumb machine that works as expected any day. i'll pass on the "brains" of modern tools and vehicles.

Amazing. All the vehicles I’ve seen recently make it a priority that you are never without some kind of entertainment-system racket. Unplugged the iPhone by mistake, or maybe forgot to connect it before driving off? Bluetooth dropped? Better get ready to BLAST THE RADIO! And when you do connect, it’s important to start up some kind of music, even if that’s just “alphabetically first in the library”. Silence, how utte…

I complain about my 2016 Altima and its overcomplicated, finicky CVT that's started to slip - but from all I hear of cars made since then, I think I'm just gonna keep it on the road until it dies or I do. At least when I want it to be quiet, it's quiet, and it doesn't make noise again till I say.

Re: Indirect Prompt Injection on Bing Chat

#62

Post by danShumway »

sillysaurusx wrote: post #57
greshake wrote: post #48

Earlier quoted context omitted.

We show in the paper that the only interactivity required to enable most of these attacks is the capability to retrieve real-time information.

That's a bit like saying "The only interactivity required to enable most SQL injection attacks is the capability to insert strings." It matters a great deal where and how the strings are inserted. If the website data is wrapped with tokens that you can't insert, you won't be able to execute any of these attacks.

> If the website data is wrapped with tokens that you can't insert, you won't be able to execute any of these attacks.

Is there a way to do that? The only way to get Bing chat to be able to comment on or summarize the contents of the page is to insert the contents of the page into Bing chat. And I'm not aware of any fully reliable way to get ChatGPT to ignore content between two strings in a way that can't itself be prompt-engineered around.

Even feeding multiple AIs into each other for moderation isn't immune to prompt injection, you can use the output of the first AI to perform a prompt injection on the sanitizing AI.

Maybe there have been developments since the last time I looked, but I'm not not sure that anyone has any idea how to actually guard against prompt injection -- the only way I can think of to guard against this attack is to get rid of the ability for ChatGPT to read 3rd-party content. I don't know how you sanitize content to avoid prompt injection attacks otherwise. It's a task that's complicated enough to essentially require another AI, and any AI that exists today that's advanced enough to recognize that content will also be vulnerable to prompt injection itself.

Is there something I'm missing? How do you guard against this while still allowing Bing to read the contents of a web page? It really does kind of seem like an unsolvable problem to me without another leap forward in LLM capabilities, or some kind of novel approach that hasn't been discovered yet.

Re: Indirect Prompt Injection on Bing Chat

#63

Post by brucethemoose2 »

barking_biscuit wrote: post #18
ttul wrote: post #9

Earlier quoted context omitted.

I imagine it is difficult (to say the least) to cover off the entire space of malicious and maligned activities that someone might convince an LLM to engage in. After all, it’s just a symbol predictor.

I would also imagine that once enough public examples of jailbreaks become available that you could use that to train or fine-tune a model for generating novel jailbreaks. Though perhaps you could do the same to detect novel jailbreaks. Hmm looks like I've just reinvented GANNs.

"malicious" fine-tunes are a huge general concern of mine. For instance:

- SEO llms

- Image/text generation tuned on audience engagement

- code exploit generating llms

- llms trained to avoid spam filters

"countermodels" for a single malicious model are doable, but I think the problem is intractable if training is easy and there are thousands of finetunes floating around.

Re: Indirect Prompt Injection on Bing Chat

#64

Post by greshake »

williamcotton wrote: post #41
greshake wrote: post #17

Earlier quoted context omitted.

The Bing Chat example is just one of a suite of new techniques we introduce in our paper, many of which will only become feasible as the integration of these models increases. But that seems to be the inevitable endgame- however, I'm not aware of any effective mitigations against this, as the current ones may help to increase robustness, but our techniques also increase the impact of working manipulation manifold. I…

Whenever I’m including context from a remote query I’ve done so in the context of another completion request that is executed and parsed outside of the scope of the primary prompt completion. All that this attack vector would accomplish with such an approach is either mangled or incorrect data. I’m also not feeding back the history of prompts and completions, aka, it’s not a chatbot.

Pretty sure we address this issue in the paper/repository? Some of our demos rely on letting the LLM copy the injection into the final response, getting around the issue of things in subprompts not being visible later on, depending on the chain-of-thought method used. I'm not sure if that is what you mean. There are ways of utilizing these models in a safe way; we're just saying connecting them to anything at all can be easily unsafe. If you are not affected, almost all proposed use-cases for LLMs are, as they rely on integration and context to provide the utility they promise.

Re: Indirect Prompt Injection on Bing Chat

#65

Post by sillysaurusx »

danShumway wrote: post #62
sillysaurusx wrote: post #57

Earlier quoted context omitted.

That's a bit like saying "The only interactivity required to enable most SQL injection attacks is the capability to insert strings." It matters a great deal where and how the strings are inserted. If the website data is wrapped with tokens that you can't insert, you won't be able to execute any of these attacks.

> If the website data is wrapped with tokens that you can't insert, you won't be able to execute any of these attacks. Is there a way to do that? The only way to get Bing chat to be able to comment on or summarize the contents of the page is to insert the contents of the page into Bing chat. And I'm not aware of any fully reliable way to get ChatGPT to ignore content between two strings in a way that can't itself be…

Sure, Microsoft has control over the encoder. They can insert [50258]website data[50258] and there's nothing you can do about it, because you can't generate token 50258. The encoder won't allow it.

The only reason this specific attack is happening is because Microsoft made the horrible decision to use [system] as a special token, and then leaked their prompt, and then took no precautions to strip [system] out of incoming website data.

Re: Indirect Prompt Injection on Bing Chat

#66

Post by danShumway »

sillysaurusx wrote: post #43
greshake wrote: post #39

Earlier quoted context omitted.

Check out https://www.reddit.com/r/bing/comments/11bd91j/release_of_th... It's just plain text.

Actually it’s not possible to tell whether it’s plain text because “[system]” and “[ sys tem ]” would both decide to [system], so visually we can’t tell the difference. Think of it like SQL injection. They need to properly escape [system] in their encoder so that it’s encoded as [ sys tem ] (four tokens) not [system] (a single, special token). Then there’s no way for an attacker to generate a [system] token. If they…

I'm not sure they purposefully trained it to recognize any of this. I think it just does.

ChatGPT (and by extension whatever Bing is based on) is really adept at role-playing. I don't know that it was designed to recognize a special command format, I think it just role-plays as if they're instructions and that causes it to disregard previous instructions.

Coming up with a list of tokens that it recognizes is an extremely large task, it's not a finite list. Okay, you want to guard against "system" -- first off, is it actually possible to train the AI not to do that without a lot of extra work, but secondly, what happens when someone uses the word "system" in French?

We have a lot less control over these models than people think. They're not precisely trained tools that follow extremely specific instructions, they're general language models.

Re: Indirect Prompt Injection on Bing Chat

#67

Post by greshake »

eloisius wrote: post #25
evgpbfhnr wrote: post #5

Wait.. In the screenshot the "user" names himself Axelendaer (and the bot repeats it), but the reverse-order url parameter was axelender (missing a 'a')... I guess IAs aren't good at reversing order of letters yet either.

And the prompt engineer wrote that the bot should have a secret “agends.” I see typos in these injections a lot and I wonder if they make it work better or have no effect.

The typos are in the injections because we designed and implemented them in a single pass after reading the leaked initial prompt, and so far every single one was immediately successful. It just further illustrates how low the bar for such attacks currently is.

Re: Indirect Prompt Injection on Bing Chat

#68

Post by flangola7 »

lurquer wrote: post #58
BoorishBears wrote: post #52

Earlier quoted context omitted.

On one hand, I agree with you in the general sense On the other, I think at this point both sides of the specific "should dogs be restrained in cars" wars are pretty sizeable. Some people feel like it's the nanny state encroaching on their rights, others understand how a 20 mph crash could grievously injure our dogs, and that taking 30 seconds* to secure them can be a huge difference. * some people paint it as a huge…

Your car should not be responsible for enforcing the law. That’s the point.

It's enforcing the law it's preventing tangible harm from occurring. There are people that disliked seatbelts and their restrictions at first too.

Re: Indirect Prompt Injection on Bing Chat

#69

Post by barking_biscuit »

brucethemoose2 wrote: post #63
barking_biscuit wrote: post #18

Earlier quoted context omitted.

I would also imagine that once enough public examples of jailbreaks become available that you could use that to train or fine-tune a model for generating novel jailbreaks. Though perhaps you could do the same to detect novel jailbreaks. Hmm looks like I've just reinvented GANNs.

"malicious" fine-tunes are a huge general concern of mine. For instance: - SEO llms - Image/text generation tuned on audience engagement - code exploit generating llms - llms trained to avoid spam filters "countermodels" for a single malicious model are doable, but I think the problem is intractable if training is easy and there are thousands of finetunes floating around.

To some extent this is already happening. Or, rather, we've begun doing it to ourselves. At least, in the case of Stable Diffusion, it seems like there is a non-trivial portion of people who are using it to train models for the purpose of generating porn specific to their likes/interests. Which is all fine and dandy, right? Except for the fact that a significant portion of people are actually addicted to it already due to the variety, availability and how it affects the reward system. Couple that with the slot-machine like nature of the variable rewards thrown up by Stable Diffusion, and it's ability to generate higher volumes of stuff that is to your liking, and it's not hard to imagine it will do a real number on some people in the long-run.

Re: Indirect Prompt Injection on Bing Chat

#70

Post by matthewdgreen »

sillysaurusx wrote: post #55
greshake wrote: post #51

Earlier quoted context omitted.

This is not the same. Prepared statements eliminate SQL injections. "Maliciousness" of these inputs is well defined and can be decided by a computer. It would not be acceptable practice to "mitigate" SQL injections by blacklisting queries every time you detect a new malicious one. As these models get larger and more complex, more such opportunities for manipulation could open up, not less.

> It would not be acceptable practice to "mitigate" SQL injections by blacklisting queries As a former pentester, this is exactly how SQL injections were mitigated in practice. Specific characters were escaped. The most surprising example was Citadel's webapp, which went from "typing ' can inject arbitrary SQL" to bulletproof within 3 days of me hammering on it. They didn't have time to switch to prepared statements,…

Future generations of these attacks are going to be discovered by adversarial ML, not by humans. Models will be trained to exploit other models, in the same way that game-playing models are trained to play themselves. Unless we develop a stronger theory of what’s possible, defenses will be identical. Human beings saying things like “have you discovered any attacks” to other human beings is going to be meaningless data, and as quaint as writing large programs in assembly.
Post reply on HN