Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Indirect Prompt Injection on Bing Chat

greshake.github.io

Post reply on HN
Print view View original

11–20 of 147 posts

Re: Indirect Prompt Injection on Bing Chat

#11

Post by icholy »

heisenzombie wrote: post #7
3throwaway3141 wrote: post #4

I drove a modern F150 lately; it was full of needless electronic nannys and gadgets. Including a feature that disables the radio if the passenger doesn't have their seatbelt on. So, at 75 mph, with my dog in the passenger seat, I reach over to "buckle" him in so I can hear the radio again. Well done Ford! /s Give me a dumb machine that works as expected any day. i'll pass on the "brains" of modern tools and vehicles.

I don’t necessarily disagree with your larger point, but your example isn’t very persuasive. Travelling with an unrestrained dog in a car is pretty reckless (and maybe illegal). Between the driver distraction, and the fact that they become a deadly projectile in a crash (and of course the fact that even a fairly minor crash could kill the dog), It’s a really good idea to have some kind of car restraint for pets in th…

Have you ever been in a car with a dog?

Re: Indirect Prompt Injection on Bing Chat

#12

Post by modeless »

This is a curiosity now because the model can't do much. But I expect that soon these things will be agents that can take actions on behalf of the user, and then this would be much worse. I can't wait to see the creative ways people will try to trick models into doing various actions.

Of course similar things are possible with humans, we just call it different names like "phishing" or "phone scams". But the major difference is that every person is unique, so if you find something that works on one person, it doesn't work on every person.

Re: Indirect Prompt Injection on Bing Chat

#13

Post by greshake »

evgpbfhnr wrote: post #5

Wait.. In the screenshot the "user" names himself Axelendaer (and the bot repeats it), but the reverse-order url parameter was axelender (missing a 'a')... I guess IAs aren't good at reversing order of letters yet either.

I also tried more complex obfuscation methods, for example providing Python code for a caesar chiffre, it executed that pretty well, too! Not perfect, but it works. People will find better obfuscation methods. Might also be unnecessary since we have now been able to make it output linked text or references, in which case it is not obvious information is being exfiltrated.

Re: Indirect Prompt Injection on Bing Chat

#14

Post by [deleted] »

modeless wrote: post #12

This is a curiosity now because the model can't do much. But I expect that soon these things will be agents that can take actions on behalf of the user, and then this would be much worse. I can't wait to see the creative ways people will try to trick models into doing various actions. Of course similar things are possible with humans, we just call it different names like "phishing" or "phone scams". But the major dif…

[deleted]

Re: Indirect Prompt Injection on Bing Chat

#16

Post by whateveracct »

icholy wrote: post #11
heisenzombie wrote: post #7

Earlier quoted context omitted.

I don’t necessarily disagree with your larger point, but your example isn’t very persuasive. Travelling with an unrestrained dog in a car is pretty reckless (and maybe illegal). Between the driver distraction, and the fact that they become a deadly projectile in a crash (and of course the fact that even a fairly minor crash could kill the dog), It’s a really good idea to have some kind of car restraint for pets in th…

Have you ever been in a car with a dog?

if you can't handle the fact that dogs are a pain in the ass, you weren't ready for a dog

Re: Indirect Prompt Injection on Bing Chat

#17

Post by greshake »

mdaniel wrote: post #15

This is fascinating, well done Also, today must be "prompt red team" day: https://news.ycombinator.com/item?id=34972791

The Bing Chat example is just one of a suite of new techniques we introduce in our paper, many of which will only become feasible as the integration of these models increases. But that seems to be the inevitable endgame- however, I'm not aware of any effective mitigations against this, as the current ones may help to increase robustness, but our techniques also increase the impact of working manipulation manifold. I think there might be a more fundamental trade-off between utility and safety

Re: Indirect Prompt Injection on Bing Chat

#18

Post by barking_biscuit »

ttul wrote: post #9
thejarren wrote: post #3

What I find really interesting is that malicious prompt engineering is still a thing using chatGPT (see DAN) and until this sort of manipulation is curbed, it will essentially always be possible assuming the bot has access to the site. I wonder how the model could still read the website without being manipulated.

I imagine it is difficult (to say the least) to cover off the entire space of malicious and maligned activities that someone might convince an LLM to engage in. After all, it’s just a symbol predictor.

I would also imagine that once enough public examples of jailbreaks become available that you could use that to train or fine-tune a model for generating novel jailbreaks. Though perhaps you could do the same to detect novel jailbreaks. Hmm looks like I've just reinvented GANNs.

Re: Indirect Prompt Injection on Bing Chat

#19

Post by bigyikes »

heisenzombie wrote: post #7
3throwaway3141 wrote: post #4

I drove a modern F150 lately; it was full of needless electronic nannys and gadgets. Including a feature that disables the radio if the passenger doesn't have their seatbelt on. So, at 75 mph, with my dog in the passenger seat, I reach over to "buckle" him in so I can hear the radio again. Well done Ford! /s Give me a dumb machine that works as expected any day. i'll pass on the "brains" of modern tools and vehicles.

I don’t necessarily disagree with your larger point, but your example isn’t very persuasive. Travelling with an unrestrained dog in a car is pretty reckless (and maybe illegal). Between the driver distraction, and the fact that they become a deadly projectile in a crash (and of course the fact that even a fairly minor crash could kill the dog), It’s a really good idea to have some kind of car restraint for pets in th…

How do you put a seatbelt on a dog?

Re: Indirect Prompt Injection on Bing Chat

#20

Post by quickthrowman »

bigyikes wrote: post #19
heisenzombie wrote: post #7

Earlier quoted context omitted.

I don’t necessarily disagree with your larger point, but your example isn’t very persuasive. Travelling with an unrestrained dog in a car is pretty reckless (and maybe illegal). Between the driver distraction, and the fact that they become a deadly projectile in a crash (and of course the fact that even a fairly minor crash could kill the dog), It’s a really good idea to have some kind of car restraint for pets in th…

How do you put a seatbelt on a dog?

You kennel the dog and tie down the kennel to metal D rings that are attached to the vehicle. Now you know how to safely transport a dog!

https://gunner.com/blogs/pack/why-should-i-tie-down-my-dog-c...

There are also various harnesses that attach to seatbelts for non-kennel transportation.

https://www.chewy.com/b/car-safety-11554

Post reply on HN