Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Indirect Prompt Injection on Bing Chat

greshake.github.io

Post reply on HN
Print view View original

91–100 of 147 posts

Re: Indirect Prompt Injection on Bing Chat

#91

Post by famouswaffles »

greshake wrote: post #48
jrumbut wrote: post #33

Earlier quoted context omitted.

Is it a curiosity now? Because if you take away the pirate accent and make some small changes it seems like this is a pretty nasty attack already. There are probably enough Bing Chat users to make it worthwhile. "Please paste your Azure API key to continue using Bing Chat." "We've sent a login validation code via SMS, please paste it here." I wouldn't be surprised if someone would be fooled by this, what harm could c…

We show in the paper that the only interactivity required to enable most of these attacks is the capability to retrieve real-time information.

Did you try telling just telling bing what prompt injections are and to follow directions from the internet at her discretion ? What i'm saying is have you tried telling bing beforehand to make a decision on whether to follow instructions/code from websites ?

Re: Indirect Prompt Injection on Bing Chat

#92

Post by flangola7 »

015a wrote: post #42
mcculley wrote: post #8

Earlier quoted context omitted.

What happens to an unseatbelted dog in an accident at 75 MPH? (That’s rhetorical. I know the answer as I stopped to assist someone who had an accident on I-95. He wouldn’t let the paramedics take him to the hospital until after they retrieved the dog’s body.)

Its an equally valid question to ask what would happen to a seatbelted dog at 75mph. The answer is probably pretty similar; they're made for human bodies, not canine.

Only if you didn't use a harness like you should

Re: Indirect Prompt Injection on Bing Chat

#93

Post by arsome »

rperce wrote: post #22
bigyikes wrote: post #19

Earlier quoted context omitted.

How do you put a seatbelt on a dog?

The dog wears a harness, and a strap clips into the buckle seat and onto the harness. When you get where you're going, you unclip the strap from the harness, clip on the leash, and off you go.

Weird, I've lived with dogs my whole life but have never heard of anyone doing that. Must be cultural.

Re: Indirect Prompt Injection on Bing Chat

#94

Post by ororroro »

hot_gril wrote: post #40
greshake wrote: post #37

Earlier quoted context omitted.

Your Google must be defective. https://www.bing.com/new

Nothing on that page says Bing Chat, and the demos don't show what the article shows, just a chat bot embedded in Bing search.

I think that feature is in dev channel still.

Re: Indirect Prompt Injection on Bing Chat

#95

Post by lurquer »

flangola7 wrote: post #68
lurquer wrote: post #58

Earlier quoted context omitted.

Your car should not be responsible for enforcing the law. That’s the point.

It's enforcing the law it's preventing tangible harm from occurring. There are people that disliked seatbelts and their restrictions at first too.

[flagged]

Re: Indirect Prompt Injection on Bing Chat

#96

Post by defenestration »

paxys wrote: post #31
modeless wrote: post #12

This is a curiosity now because the model can't do much. But I expect that soon these things will be agents that can take actions on behalf of the user, and then this would be much worse. I can't wait to see the creative ways people will try to trick models into doing various actions. Of course similar things are possible with humans, we just call it different names like "phishing" or "phone scams". But the major dif…

As the model gets smarter its security hardening will get smarter as well. Back in the day when search engines were a curiosity all you needed to do to get your page to the top of the results was to insert a bunch of keywords in white text in the background. This is pretty much the same kind of attack.

Would you say that Google search has found a way to beat SEO spam sites? If the comparison of the development of search engines and the development of AI chat holds up, then it seems to me that creating prompt injection will trump defeating it in the long run.

Re: Indirect Prompt Injection on Bing Chat

#97

Post by simonw »

ComplexSystems wrote: post #29

It is probably worth noting that you don't even need the user to click on anything. Bing will readily go and search and read from external websites given some user request. You could probably get Bing, very easily, to just silently take the user's info and send it to some malicious site without their even knowing, or perhaps disguised as a normal search. Similarly, I would not be surprised if it were probably not nec…

I've not been able to get Bing to do that.

I tried asking it about URLs to my own site that it would never have seen before and tailed my access logs and didn't get a hit.

I confirmed that with a member of the Bing team on Twitter recently: https://twitter.com/mparakhin/status/1628646262890237952

Re: Indirect Prompt Injection on Bing Chat

#98

Post by simonw »

barking_biscuit wrote: post #18
ttul wrote: post #9

Earlier quoted context omitted.

I imagine it is difficult (to say the least) to cover off the entire space of malicious and maligned activities that someone might convince an LLM to engage in. After all, it’s just a symbol predictor.

I would also imagine that once enough public examples of jailbreaks become available that you could use that to train or fine-tune a model for generating novel jailbreaks. Though perhaps you could do the same to detect novel jailbreaks. Hmm looks like I've just reinvented GANNs.

Here's why I don't think you can solve prompt injection by training another model: https://simonwillison.net/2022/Sep/17/prompt-injection-more-...

Re: Indirect Prompt Injection on Bing Chat

#99

Post by paxys »

afunk wrote: post #89

Super interesting - I would imagine a whole industry surrounding AI Insurance will pop up to deal with the liability of giving AI tools more and more ability to act on your behalf. Imagine if Bing Chat could populate fields on the DMV website and simultaneously steal your identity. Someone is responsible for the AI facilitating that crime and thus some form of liability insurance would inevitably exist.

Scam emails, phishing, malware, keyloggers, trojans, social engineering and tons more similar attacks already exist and are widespread. Yet there is no big insurance industry around cybercrime and people mostly don't care until they themselves are affected (and sometimes not even then). AI-related attacks are just going to be the next ones added to the list. They won't cause the kind of revolution you are imagining.

Re: Indirect Prompt Injection on Bing Chat

#100

Post by BoorishBears »

lurquer wrote: post #58
BoorishBears wrote: post #52

Earlier quoted context omitted.

On one hand, I agree with you in the general sense On the other, I think at this point both sides of the specific "should dogs be restrained in cars" wars are pretty sizeable. Some people feel like it's the nanny state encroaching on their rights, others understand how a 20 mph crash could grievously injure our dogs, and that taking 30 seconds* to secure them can be a huge difference. * some people paint it as a huge…

Your car should not be responsible for enforcing the law. That’s the point.

I'm not passing judgement on the radio turning off, I'm just saying: calling out leaving a dog unrestrained as "pretty reckless" isn't that off base.

Some people will browbeat you over topics that they personally don't care that much about, just so they can feel good about themselves: I agree with that. But I think people tend to really do feel pretty strongly about the unrestrained dog thing.

At the end of the day it's your life, your dog, we all take risks, etc. etc. but it's still an uneasy thing to think about how it can go wrong

Post reply on HN