Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: Is “prompt injection” going to be a new common vulnerability?

news.ycombinator.com

Post reply on HN
Print view View original

21–30 of 116 posts

Re: Ask HN: Is “prompt injection” going to be a new common vulnerability?

#21

Post by nzoschke »

UncleEntity wrote: post #15
kadoban wrote: post #7

Earlier quoted context omitted.

The way forward eventually is going to be to just not bother with any of this crap, and let it run free. The tech exists, and the problematic outputs are what the user says they want, eventually they're going to win out.

They’re not going to let it run free or you will see countless articles on “ChatGPT is a Holocaust denier, news at 11”. And the lawsuits, oh the lawsuits. ChatGPT convinced my daughter to join a cult and now is a child bride, honest, Your Honor.

I think you’re both right. Microsoft won’t let theirs run free but there will be other vendors that do.

Who is intimately responsible for all of this?

Is it the end user? Don’t ask questions you don’t want to hear potentially dangerous answers to.

Is it Microsoft? It’s their product.

Is it OpenAI as Microsoft’s vendor?

When we start plugging in the moderation AI is it their responsibility for things that slip through?

Who and where did they get their training data from? And is there any ability to attribute things back to specific sources of training data and blame and block them?

Lots of layers. Little to no humans directly responsible for what it decides to say.

Maybe the end user does have to deal with it…

Re: Ask HN: Is “prompt injection” going to be a new common vulnerability?

#22

Post by SheinhardtWigCo »

There's a spectrum between not using LLMs at all, and exposing their output directly to the user.

I think the most successful programs to leverage LLMs will be ones that use the model's output to be better or more intuitive in some way, optimistically, without exposing completion text directly in the UI.

Re: Ask HN: Is “prompt injection” going to be a new common vulnerability?

#23

Post by roxgib »

UncleEntity wrote: post #15
kadoban wrote: post #7

Earlier quoted context omitted.

The way forward eventually is going to be to just not bother with any of this crap, and let it run free. The tech exists, and the problematic outputs are what the user says they want, eventually they're going to win out.

They’re not going to let it run free or you will see countless articles on “ChatGPT is a Holocaust denier, news at 11”. And the lawsuits, oh the lawsuits. ChatGPT convinced my daughter to join a cult and now is a child bride, honest, Your Honor.

We used to see those articles, but now that the models are actually good enough to be useful I think people are much more willing to overlook the flaws.

Re: Ask HN: Is “prompt injection” going to be a new common vulnerability?

#24

Post by nonethewiser »

UncleEntity wrote: post #15
kadoban wrote: post #7

Earlier quoted context omitted.

The way forward eventually is going to be to just not bother with any of this crap, and let it run free. The tech exists, and the problematic outputs are what the user says they want, eventually they're going to win out.

They’re not going to let it run free or you will see countless articles on “ChatGPT is a Holocaust denier, news at 11”. And the lawsuits, oh the lawsuits. ChatGPT convinced my daughter to join a cult and now is a child bride, honest, Your Honor.

> They’re not going to let it run free or you will see countless articles on “ChatGPT is a Holocaust denier, news at 11”.

If we're afraid of that then we're already worse off.

Re: Ask HN: Is “prompt injection” going to be a new common vulnerability?

#25

Post by reilly3000 »

Waterluvian wrote: post #14
brrsty wrote: post #12

Earlier quoted context omitted.

What's worse - people tricking AI, or AI tricking people?

I guess it depends on whose side you’re on.

I am a historian who was born in 3000 ‘AD’ as you say, and I am curious about your thoughts. What’s the best way you can describe the ‘sides’ your response alludes at? How would you name these divergences?

Re: Ask HN: Is “prompt injection” going to be a new common vulnerability?

#26

Post by PaulDavisThe1st »

brrsty wrote: post #12
Waterluvian wrote: post #11

Does anyone else feel kind of wowed by how this technology’s exploits are also quite similar to a human? You can kind of trick it into divulging information not meant for you by somehow “persuading” it to tell you. It didn’t want to tell me how to do something unethical until I said, “well, it’s for a school play.” It’s like the thing is born yesterday. It’s intelligent but it has no street smarts. It can be fooled e…

What's worse - people tricking AI, or AI tricking people?

or ... people tricking AI into tricking people?

or ... ... AI tricking people into tricking people?

Re: Ask HN: Is “prompt injection” going to be a new common vulnerability?

#27

Post by totetsu »

reilly3000 wrote: post #25
Waterluvian wrote: post #14

Earlier quoted context omitted.

I guess it depends on whose side you’re on.

I am a historian who was born in 3000 ‘AD’ as you say, and I am curious about your thoughts. What’s the best way you can describe the ‘sides’ your response alludes at? How would you name these divergences?

human-centered" and "AI-centered"

Re: Ask HN: Is “prompt injection” going to be a new common vulnerability?

#28

Post by siliconc0w »

It'll be kinda funny when these become like zero day exploits to get the AIs to slip up and stray outside their sanitized space. I suspect when they're more powerful they'll sanitize not just politically correct areas but financial analysis or other topics that could be especially valuable and sold under a higher "premium" tier...

Re: Ask HN: Is “prompt injection” going to be a new common vulnerability?

#30

Post by simonw »

Yes. Prompt injection will continue to be a common vulnerability for quite a while, from what I've seen.

I wrote a bunch about this back in September:

- https://simonwillison.net/2022/Sep/12/prompt-injection/ was I believe the first blog entry to use the term "prompt injection"

- https://simonwillison.net/2022/Sep/16/prompt-injection-solut... - "I don't know how to solve prompt injection" - talks about how, unlike attacks like SQL injection, I don't actually know of a guaranteed mitigation for this class of attack

- https://simonwillison.net/2022/Sep/17/prompt-injection-more-... - "You can’t solve AI security problems with more AI" is my argument that using more prompt engineering to do things like detect if an incoming prompt contains an injection attack isn't very likely to work

It's five months later now and I am yet to be convinced that there's an easy fix to this problem.

Microsoft's new Bing Chatbot is vulnerable to a prompt leak attack - and Microsoft worked with OpenAI directly on building that! https://twitter.com/kliu128/status/1623472922374574080

Post reply on HN