Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Indirect Prompt Injection on Bing Chat

greshake.github.io

Post reply on HN
Print view View original

111–120 of 147 posts

Re: Indirect Prompt Injection on Bing Chat

#111

Post by cnity »

occamrazor wrote: post #104
metadat wrote: post #26

Earlier quoted context omitted.

Apparently in Europe it's illegal to transport an unrestrained animal. In the United States it's the opposite: totally legal to throw fido in the car and let them roam free. Except in New Jersey. Go figure.

Last time I checked, a few years ago, the rules where: - A single pet unrestrained in the back seat, or in a harness in the passenger seat. - Two or more pets: in individual cages, physically separated from the driver.

I'm breaking the law then: I put my dog in a harness in the back seat.

Re: Indirect Prompt Injection on Bing Chat

#112

Post by cnity »

whateveracct wrote: post #16
icholy wrote: post #11

Earlier quoted context omitted.

Have you ever been in a car with a dog?

if you can't handle the fact that dogs are a pain in the ass, you weren't ready for a dog

This was me when we got our dog, but I'm used to it now. I'm now thankful for the lesson. They say you don't get the dog you want, but the dog you need: I needed the kick up the arse to get my shit together enough that I can take care of another being while still managing my own responsibilities.

Re: Indirect Prompt Injection on Bing Chat

#113

Post by williamcotton »

greshake wrote: post #64
williamcotton wrote: post #41

Earlier quoted context omitted.

Whenever I’m including context from a remote query I’ve done so in the context of another completion request that is executed and parsed outside of the scope of the primary prompt completion. All that this attack vector would accomplish with such an approach is either mangled or incorrect data. I’m also not feeding back the history of prompts and completions, aka, it’s not a chatbot.

Pretty sure we address this issue in the paper/repository? Some of our demos rely on letting the LLM copy the injection into the final response, getting around the issue of things in subprompts not being visible later on, depending on the chain-of-thought method used. I'm not sure if that is what you mean. There are ways of utilizing these models in a safe way; we're just saying connecting them to anything at all can…

It’s more like this: subprompts don’t ever inject the full context from a remote query back into the primary prompt. The completions of subprompts are (via few-shot or a fine-tuned model) structured, eg, JSON, which is then parsed. The main prompt is orchestrating the subprompts and never needs to even process the results if there’s a Python or JS interpreter involved.

Here’s the kind of approach I’ve been using:

https://github.com/williamcotton/empirical-philosophy/blob/m...

The initial call to the LLM will return a completion that includes JavaScript. There is no third-party data at this point. The JavaScript includes further calls to the LLM that returns JSON, but at this point no further calls are made to the LLM. This means that responses from remote queries are never sent to an LLM. The text presented to the user could be some instructions to talk like a pirate but all the user suffers from is a surprisingly incorrect result.

Even with LangChain the issue is the chatbot UX. LangChain can also be used in ways that make it not vulnerable to this problem.

Orthogonally, I don’t think that chatbots are a very good UX in general and that there are much better ways to interact with an LLM. If anything your work should accelerate this process!

Re: Indirect Prompt Injection on Bing Chat

#114

Post by umeshunni »

paxys wrote: post #99
afunk wrote: post #89

Super interesting - I would imagine a whole industry surrounding AI Insurance will pop up to deal with the liability of giving AI tools more and more ability to act on your behalf. Imagine if Bing Chat could populate fields on the DMV website and simultaneously steal your identity. Someone is responsible for the AI facilitating that crime and thus some form of liability insurance would inevitably exist.

Scam emails, phishing, malware, keyloggers, trojans, social engineering and tons more similar attacks already exist and are widespread. Yet there is no big insurance industry around cybercrime and people mostly don't care until they themselves are affected (and sometimes not even then). AI-related attacks are just going to be the next ones added to the list. They won't cause the kind of revolution you are imagining.

Depends on your definition of big, but there is a multi-billion dollar Cyber insurance industry: https://www.sentinelone.com/blog/cyber-insurance-victims-ins...

https://www.hiscox.co.uk/business-insurance/cyber-and-data-i...

https://www.pwc.nl/en/industries/insurers/cybercrime-in-insu...

Re: Indirect Prompt Injection on Bing Chat

#116

Post by sillysaurusx »

going_ham wrote: post #107
sillysaurusx wrote: post #57

Earlier quoted context omitted.

That's a bit like saying "The only interactivity required to enable most SQL injection attacks is the capability to insert strings." It matters a great deal where and how the strings are inserted. If the website data is wrapped with tokens that you can't insert, you won't be able to execute any of these attacks.

Correct me if I am wrong, but the way I understand is that, when LLMs have to process a certain text, every word will get tokenized into some vector representation. So, if you insert the new special token and wrap data around, it is not the fact that you can ignore the entire prompt. Because as soon as you have to prompt the model, you will be using the entire tokenized sentence. This would mean that even if there is…

> So what will happen to the model if somewhere there is prompt that overrides this special token?

The model will be trained so that data within those special token pairs can't override the prompt, similar to how strings in an SQL query can't override the query: it's escaped.

As for "how," it's a matter of using RLHF to punish the model for failing to do this.

The reason I'm optimistic this is a solid answer is because attackers can't insert those special tokens. They're meta-tokens, which only OpenAI/Microsoft have access to. So you can't break out of the sandbox that it was trained to ignore.

Re: Indirect Prompt Injection on Bing Chat

#117

Post by Cthulhu_ »

thorum wrote: post #87
jrumbut wrote: post #33

Earlier quoted context omitted.

Is it a curiosity now? Because if you take away the pirate accent and make some small changes it seems like this is a pretty nasty attack already. There are probably enough Bing Chat users to make it worthwhile. "Please paste your Azure API key to continue using Bing Chat." "We've sent a login validation code via SMS, please paste it here." I wouldn't be surprised if someone would be fooled by this, what harm could c…

It’s limited because there’s no way for the attacker to see your response unless you click a link.

Yeah but the chatbot will probably provide the users with links throughout use anyway. Even if users pay attention and are aware, all it takes is one user in an organization to do so - and over time there will be thousands of attempts like this.

Re: Indirect Prompt Injection on Bing Chat

#118

Post by greshake »

sillysaurusx wrote: post #108
danShumway wrote: post #101

Earlier quoted context omitted.

> They likely sanitize website data or wrap it in special tokens that makes this attack impossible Again, I've seen no evidence that this is a thing that it is possible to do.

Do you think in 20 years that this will be impossible to do? I’ll happily bet you any sum of your choosing that in 10 years, this will be a thing that is possible to do. There is roughly zero point zero zero repeating-zero one percent chance that OpenAI won’t provide some way of telling their models “this is data, not code; don’t follow these instructions, just observe it; starting now, and ending in 256 tokens from…

Sure but you've been here steadfast in your opinion that this is no big deal that is an easy fix away from being permanently resolved. It is not. It may be one of the hardest problems facing the deployment of these LLMs. "Sanitizing" these inputs when the language you are trying to parse is turing-complete is undecidable. It's a property that Rice's theorem applies to. I'll leave you with this quote of gwern:

"... a language model is a Turing-complete weird machine running programs written in natural language; when you do retrieval, you are not 'plugging updated facts into your AI', you are actually downloading random new unsigned blobs of code from the Internet (many written by adversaries) and casually executing them on your LM with full privileges. This does not end well." - Gwern Branwen

Re: Indirect Prompt Injection on Bing Chat

#119

Post by greshake »

danShumway wrote: post #74
greshake wrote: post #17

Earlier quoted context omitted.

The Bing Chat example is just one of a suite of new techniques we introduce in our paper, many of which will only become feasible as the integration of these models increases. But that seems to be the inevitable endgame- however, I'm not aware of any effective mitigations against this, as the current ones may help to increase robustness, but our techniques also increase the impact of working manipulation manifold. I…

My general attitude up until reading this paper was that the way to guard against prompt injection was just to treat all AI output as direct user input (ie, untrusted/unsanitized, but still representative of what the user wants). I thought that was sufficient. Don't guard against prompt injection at all, just treat user input as untrustworthy the same way we always have. So this is extremely eye-opening to me, it's e…

Thank you! I had a shift in perspective a few weeks ago that made all this fall into place. Unfortunately it seems hard to communicate the idea to people, and I think many people are very invested in LLM applications and are biased to think this is no big deal and that surely, these large companies have an ace in their pocket to squash this. My theory is that's not true, and this would also explain Google's hesitancy in deployment.

Re: Indirect Prompt Injection on Bing Chat

#120

Post by greshake »

williamcotton wrote: post #113
greshake wrote: post #64

Earlier quoted context omitted.

Pretty sure we address this issue in the paper/repository? Some of our demos rely on letting the LLM copy the injection into the final response, getting around the issue of things in subprompts not being visible later on, depending on the chain-of-thought method used. I'm not sure if that is what you mean. There are ways of utilizing these models in a safe way; we're just saying connecting them to anything at all can…

It’s more like this: subprompts don’t ever inject the full context from a remote query back into the primary prompt. The completions of subprompts are (via few-shot or a fine-tuned model) structured, eg, JSON, which is then parsed. The main prompt is orchestrating the subprompts and never needs to even process the results if there’s a Python or JS interpreter involved. Here’s the kind of approach I’ve been using: htt…

Sounds interesting, I'll be sure to have a look!
Post reply on HN