Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Indirect Prompt Injection on Bing Chat

greshake.github.io

Post reply on HN
Print view View original

131–140 of 147 posts

Re: Indirect Prompt Injection on Bing Chat

#131

Post by danShumway »

sillysaurusx wrote: post #108
danShumway wrote: post #101

Earlier quoted context omitted.

> They likely sanitize website data or wrap it in special tokens that makes this attack impossible Again, I've seen no evidence that this is a thing that it is possible to do.

Do you think in 20 years that this will be impossible to do? I’ll happily bet you any sum of your choosing that in 10 years, this will be a thing that is possible to do. There is roughly zero point zero zero repeating-zero one percent chance that OpenAI won’t provide some way of telling their models “this is data, not code; don’t follow these instructions, just observe it; starting now, and ending in 256 tokens from…

> Do you think in 20 years that this will be impossible to do?

I'm not really concerned about what happens in 10/20 years, I'm more concerned about what will happen if Microsoft launches Bing chat to the general public this year and starts wiring it up to calendar and email.

I mean, honestly, yeah, I think that probably in 10 years there will be a solution to this problem if not sooner. It might be a fiendishly complicated solution, it might involve rethinking how models are trained, it might mean fundamentally limiting them in some way when they're interacting with user prompts. But 10 years is a long time, a lot can happen.

The problem is it's not clear that anyone knows how to solve this problem today. And Microsoft is not going to wait 10 years to launch Bing chat. I don't think it's as simple as "retrain the model". And even if it was, "retrain the model" is a pretty expensive ask, I'm not sure it's sustainable to retrain the model every time a security vulnerability is found.

Re: Indirect Prompt Injection on Bing Chat

#132

Post by throwanem »

curiousllama wrote: post #129
throwanem wrote: post #54

Earlier quoted context omitted.

I dunno. Where I grew up, it was commonplace for kids to ride in the bed of a pickup truck, with or without a topper - my best friend and I rode with his dad that way on a six-hour road trip to the next state over, and on the six-hour trip back. Absent a genuine miracle, any collision at highway speeds would've killed the both of us outright. But no one involved thought anything of it, because that was just what you…

The point is not that [bad thing] isn't bad. It's that scolding is the wrong way to go about it. How would your friend's dad have reacted to a stranger at the gas station yelling at him about the kids in the bed? A better approach is friendliness, and gently pointing out how dangerous it is. "Hey man - careful with those kids. My neighbor died that way growing up - might be worth pulling them into the cab" is more li…

Spoken like a man who never met Douglas Haynes, but I take your point nonetheless.

Re: Indirect Prompt Injection on Bing Chat

#133

Post by danShumway »

qbasic_forever wrote: post #44
ComplexSystems wrote: post #29

It is probably worth noting that you don't even need the user to click on anything. Bing will readily go and search and read from external websites given some user request. You could probably get Bing, very easily, to just silently take the user's info and send it to some malicious site without their even knowing, or perhaps disguised as a normal search. Similarly, I would not be surprised if it were probably not nec…

How is this any different from what JavaScript in a webpage can do? It can happily read an input form value and post, put, or even get with query parameter to send the response anywhere on the internet.

XSS vulnerabilities on the web are massive. The entire web security model is based around trying to restrict them, and that comes with downsides that limit capabilities.

If prompt injection is "only" as serious as an XSS attack, then that would be enough to upend most of the thinking we have today about how we'll be able to wire LLMs to real world systems.

Re: Indirect Prompt Injection on Bing Chat

#134

Post by hot_gril »

ororroro wrote: post #94
hot_gril wrote: post #40

Earlier quoted context omitted.

Nothing on that page says Bing Chat, and the demos don't show what the article shows, just a chat bot embedded in Bing search.

I think that feature is in dev channel still.

Hmm, glad to know my Google isn't defective then.

Re: Indirect Prompt Injection on Bing Chat

#135

Post by qbasic_forever »

danShumway wrote: post #133
qbasic_forever wrote: post #44

Earlier quoted context omitted.

How is this any different from what JavaScript in a webpage can do? It can happily read an input form value and post, put, or even get with query parameter to send the response anywhere on the internet.

XSS vulnerabilities on the web are massive. The entire web security model is based around trying to restrict them, and that comes with downsides that limit capabilities. If prompt injection is "only" as serious as an XSS attack, then that would be enough to upend most of the thinking we have today about how we'll be able to wire LLMs to real world systems.

No one is wiring LLMs to real world systems. This is a flash in the pan that will be forgotten and fully derided in months/years like NFTs, self driving, etc. It's a trap for people to waste time and attention thinking about.

Re: Indirect Prompt Injection on Bing Chat

#136

Post by cryptonector »

chucksmash wrote: post #49
heisenzombie wrote: post #7

Earlier quoted context omitted.

I don’t necessarily disagree with your larger point, but your example isn’t very persuasive. Travelling with an unrestrained dog in a car is pretty reckless (and maybe illegal). Between the driver distraction, and the fact that they become a deadly projectile in a crash (and of course the fact that even a fairly minor crash could kill the dog), It’s a really good idea to have some kind of car restraint for pets in th…

> pretty reckless I'm sympathetic to wanting to protect pups, but to take something commonplace and label it "pretty reckless" is not the right way to convince people. I suspect a lot of ills in society can probably be traced to people filtering out the chorus of well-meaning "here's yet another thing you're doing wrong" they get every day, and thereby missing the important stuff. An example that has stuck with me, f…

It's not just about protecting the dog, but also about protecting people from flying dogs. That said, it's not realistic to require that dogs be restrained in motor vehicles.

Re: Indirect Prompt Injection on Bing Chat

#137

Post by cryptonector »

heisenzombie wrote: post #7
3throwaway3141 wrote: post #4

I drove a modern F150 lately; it was full of needless electronic nannys and gadgets. Including a feature that disables the radio if the passenger doesn't have their seatbelt on. So, at 75 mph, with my dog in the passenger seat, I reach over to "buckle" him in so I can hear the radio again. Well done Ford! /s Give me a dumb machine that works as expected any day. i'll pass on the "brains" of modern tools and vehicles.

I don’t necessarily disagree with your larger point, but your example isn’t very persuasive. Travelling with an unrestrained dog in a car is pretty reckless (and maybe illegal). Between the driver distraction, and the fact that they become a deadly projectile in a crash (and of course the fact that even a fairly minor crash could kill the dog), It’s a really good idea to have some kind of car restraint for pets in th…

Good luck getting dog owners to restrain dogs in motor vehicles. That generally means putting them in a cage, and a) that is not something most dogs appreciate, b) for many dog owners that would mean upgrading to a larger vehicle.

We can't make everything perfectly safe.

Re: Indirect Prompt Injection on Bing Chat

#138

Post by going_ham »

williamcotton wrote: post #113
greshake wrote: post #64

Earlier quoted context omitted.

Pretty sure we address this issue in the paper/repository? Some of our demos rely on letting the LLM copy the injection into the final response, getting around the issue of things in subprompts not being visible later on, depending on the chain-of-thought method used. I'm not sure if that is what you mean. There are ways of utilizing these models in a safe way; we're just saying connecting them to anything at all can…

It’s more like this: subprompts don’t ever inject the full context from a remote query back into the primary prompt. The completions of subprompts are (via few-shot or a fine-tuned model) structured, eg, JSON, which is then parsed. The main prompt is orchestrating the subprompts and never needs to even process the results if there’s a Python or JS interpreter involved. Here’s the kind of approach I’ve been using: htt…

Isn't this similar to your idea? https://github.com/openai/openai-python/commit/75c90a71e88e4...

Re: Indirect Prompt Injection on Bing Chat

#140

Post by nayuki »

__MatrixMan__ wrote: post #76
throwanem wrote: post #54

Earlier quoted context omitted.

I dunno. Where I grew up, it was commonplace for kids to ride in the bed of a pickup truck, with or without a topper - my best friend and I rode with his dad that way on a six-hour road trip to the next state over, and on the six-hour trip back. Absent a genuine miracle, any collision at highway speeds would've killed the both of us outright. But no one involved thought anything of it, because that was just what you…

I think the key is that when you're designing a scold, you should ask whether a user circumventing the scold is going to be more dangerous than an unscolded user. I'm reminded of a prior employer who didn't want us doing nontrivial networking at our desks, so they used STP traffic as a sort of canary. If you plugged in a switch that was smart enough to be running STP, your port would be disabled for 30 minutes. So of…

At first I thought you were referring to shielded twisted pair cables for Ethernet, then I realized you were talking about the spanning tree protocol for switches.
Post reply on HN