Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: Is “prompt injection” going to be a new common vulnerability?

news.ycombinator.com

Post reply on HN
Print view View original

71–80 of 116 posts

Re: Ask HN: Is “prompt injection” going to be a new common vulnerability?

#71

Post by simonw »

bawolff wrote: post #49

A vulnerability? Yes. A serious one... i don't really think so in the grand scheme of things. Injection vulnerabilities in one form or another are like 90% of all security vulnerabilities. We have the obvious ones like sql injection or shell injection. We dont call XSS injection but it really is just html/js injection. Even things like buffer overflows are injections if viewed through the right lens. If there is one…

There's a crucial difference between prompt injection and other injection attacks such as SQL injection or XSS or shell injection.

For all of those other injection attacks we know what the mitigations are: parameterized queries for SQL injection. Context-aware HTML escaping for XSS. Shell special character escaping for shell commands.

Prompt injection does not have a reliable mitigation yet. It's currently an injection attack without a fix.

Re: Ask HN: Is “prompt injection” going to be a new common vulnerability?

#72

Post by wnkrshm »

im3w1l wrote: post #55
chowells wrote: post #48

Earlier quoted context omitted.

Yes, there are absolutely dangers. It shouldn't be possible for a depressed person to convince a chatbot to tell them to commit suicide. There are some people who only need the tiniest push on a bad day.

Why on earth would this depressed person use prompt injections to force the bot to give bad advice though?

In many countries websites describing ways to commit suicide or some people's public 'goodbye letters' explaining their reasons for suicide are removed by law enforcement, when they are discovered.

Since you can ask ChatGPT what the most painless and direct way to kill yourself is, should chatGPT be able to assist you in planning your suicide? (Edit: i.e. having a chat partner that directly gives you feedback on ideas and does not try to talk you down/away from them)

While suicide isn't illegal, in many countries helping someone commit suicide is a capital crime.

Re: Ask HN: Is “prompt injection” going to be a new common vulnerability?

#74

Post by birdyrooster »

nhchris wrote: post #45
andrewmcwatters wrote: post #3

If I understand correctly, ChatGPT doesn't have its latent capabilities removed. Instead, they're suppressed by training using negative feedback. These special prompts are supposed to find the remaining stochastic spaces where ChatGPT can process the desired output that is not suppressed by training. So, the danger seems to be that there is no currently documented way to completely remove these possible outputs, beca…

> the danger seems to be that there is no currently documented way to completely remove these possible outputs I know OpenAI likes throwing around terms like "danger" and "harm" liberally, but is this really a danger ? Outside of hypothetical scenarios where someone wires ChatGPT to a self-driving trolley.

It won’t even tell me strategies for video games which are violent. It can’t help me to understand violence as a concept or how it is used so that I can counteract it. It’s so nerfed its pathetic.

Re: Ask HN: Is “prompt injection” going to be a new common vulnerability?

#75

Post by wnkrshm »

graypegg wrote: post #6
andrewmcwatters wrote: post #3

If I understand correctly, ChatGPT doesn't have its latent capabilities removed. Instead, they're suppressed by training using negative feedback. These special prompts are supposed to find the remaining stochastic spaces where ChatGPT can process the desired output that is not suppressed by training. So, the danger seems to be that there is no currently documented way to completely remove these possible outputs, beca…

Oh that’s interesting! I was just assuming they determined some large list of words and phrases that you couldn’t use. Makes a lot more sense to front-load a bunch of prompt data about “chat bot does not discuss politicians”. I guess this evolves into an arms race where we’re constantly growing that initial prompt.

OpenAI used labor in developing countries for sifting through illegal outputs of the model, to train against that [0].

[0] https://time.com/6247678/openai-chatgpt-kenya-workers/

Re: Ask HN: Is “prompt injection” going to be a new common vulnerability?

#76

Post by simonw »

alex_duf wrote: post #70
simonw wrote: post #65

Earlier quoted context omitted.

It looks like this is almost impossibly difficult to actually implement against existing large language models: it's been at least five months since people started talking about this solution and so far no one has managed to deliver a working implementation.

5 months is nothing, we're still in the infancy of the technology, I think this wasn't a priority until now. Give it a few years, a few research papers and I'm sure someone will figure it out.

Yeah I'd be surprised if this wasn't figured out in a few years time too... but with the rate at which these systems are being built and deployed right now we could really do with a solution earlier than that!

Re: Ask HN: Is “prompt injection” going to be a new common vulnerability?

#77

Post by wnkrshm »

deafpolygon wrote: post #53
_nalply wrote: post #51

A solution might be: use two different AIs. The first one you can prompt to your heart's content. The second one is never prompted by anyone except the service provider. The second one does the filtering.

Should we call the second AI Conscience?

Super-ego

Re: Ask HN: Is “prompt injection” going to be a new common vulnerability?

#78

Post by somenameforme »

Waterluvian wrote: post #11

Does anyone else feel kind of wowed by how this technology’s exploits are also quite similar to a human? You can kind of trick it into divulging information not meant for you by somehow “persuading” it to tell you. It didn’t want to tell me how to do something unethical until I said, “well, it’s for a school play.” It’s like the thing is born yesterday. It’s intelligent but it has no street smarts. It can be fooled e…

IMO a lot of what we're seeing and inferring is an optical illusion of sorts. We've created is a natural language interface. And that is a huge accomplishment, but it can also make one see things which are not necessarily there. Imagine a primitive natural language interface for your console:

- You: "Show me all files."

- Com: [Outputs a list of files excepting hidden]

- You: "I said all files."

- Com: "I did show you all files."

- You: "Including hidden."

- Com: "Oh, OK." [Outputs all files]

You're not really tricking it, so much as effortfully changing your "ls" to an "ls -al". But the interface would make it feel like you're interacting with an intelligent system, and maybe even getting it to do something it shouldn't. This is made even more extreme right now given that the state of the art in access control seems to be to name your "secure" directory ¶. Nobody will ever figure out how to access that!

Re: Ask HN: Is “prompt injection” going to be a new common vulnerability?

#79

Post by _nalply »

deafpolygon wrote: post #73
Nition wrote: post #60

Earlier quoted context omitted.

We shall call it Amygdata.

Yes, Lieutenant Amygdata. This is the way.

A Deaf lieutnant! If you are really Deaf, dear deafpolygon, please contact me. See my profile, there is a (hidden) way to contact me.

Re: Ask HN: Is “prompt injection” going to be a new common vulnerability?

#80

Post by int_19h »

_nalply wrote: post #51

A solution might be: use two different AIs. The first one you can prompt to your heart's content. The second one is never prompted by anyone except the service provider. The second one does the filtering.

If it's filtering by taking the output of the first model as a prompt (with some framing), then that is equally susceptible to prompt engineering. Indeed, you can already tell ChatGPT to write a prompt for itself to do such and such, and it will do so. You can even tell it to write a prompt to write a prompt.
Post reply on HN