Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: Is “prompt injection” going to be a new common vulnerability?

news.ycombinator.com

Post reply on HN
Print view View original

41–50 of 116 posts

Re: Ask HN: Is “prompt injection” going to be a new common vulnerability?

#41

Post by simonw »

theamk wrote: post #8

I think this is a vulnerability in the sense that ability to "View Source" is vulnerability. Some technologies allow users to see the source code. They just work like this. Programmer should be aware of it and should not put any confidential information there.

There was a report just a few days ago of a system that was passing output to the Python eval() function - someone used that to steal an OpenAI API key: https://twitter.com/ludwig_stumpp/status/1619701277419794435

It's vitally important that anyone building against language models like GPT3 understands prompt injection in depth, so they don't make mistakes like this.

Re: Ask HN: Is “prompt injection” going to be a new common vulnerability?

#42

Post by jhoelzel »

IMHO Prompt-, like SQL-Injection will largely be used to steal prompts, so therefore the "business model" of some startups and will largely be automated.

And its even worse: where sql at least requires "some" knowledge of the database below, prompt injection will just flat out work over all "chat like bots".

Its not inherently a problem, but the more functionality you give to your bot the more it can be exploited and I do see DDOS attacks by chat bots as a very real possebility.

Re: Ask HN: Is “prompt injection” going to be a new common vulnerability?

#43

Post by luckylion »

theamk wrote: post #8

I think this is a vulnerability in the sense that ability to "View Source" is vulnerability. Some technologies allow users to see the source code. They just work like this. Programmer should be aware of it and should not put any confidential information there.

If you're OpenAI, you couldn't really care less about it, I suppose, because people need you to run those prompts, they're not easily transferred between models if I understand correctly.

But if you're an OpenAI-API-Reseller and your value proposition is prepending a prompt to whatever input you're given, that's very much a concern, because people can easily cut out the middle man if they have the prompt. The SaaS boom has happened for the same reason, hasn't it? If you deliver a software / library, people can look at it and replace you. If all you provide an API where nobody can "view source", they can't, and you can forever collect rent.

Re: Ask HN: Is “prompt injection” going to be a new common vulnerability?

#44

Post by luckylion »

simonw wrote: post #41
theamk wrote: post #8

I think this is a vulnerability in the sense that ability to "View Source" is vulnerability. Some technologies allow users to see the source code. They just work like this. Programmer should be aware of it and should not put any confidential information there.

There was a report just a few days ago of a system that was passing output to the Python eval() function - someone used that to steal an OpenAI API key: https://twitter.com/ludwig_stumpp/status/1619701277419794435 It's vitally important that anyone building against language models like GPT3 understands prompt injection in depth, so they don't make mistakes like this.

API response => eval() it's going to be an interesting future.

Re: Ask HN: Is “prompt injection” going to be a new common vulnerability?

#45

Post by nhchris »

andrewmcwatters wrote: post #3

If I understand correctly, ChatGPT doesn't have its latent capabilities removed. Instead, they're suppressed by training using negative feedback. These special prompts are supposed to find the remaining stochastic spaces where ChatGPT can process the desired output that is not suppressed by training. So, the danger seems to be that there is no currently documented way to completely remove these possible outputs, beca…

> the danger seems to be that there is no currently documented way to completely remove these possible outputs

I know OpenAI likes throwing around terms like "danger" and "harm" liberally, but is this really a danger? Outside of hypothetical scenarios where someone wires ChatGPT to a self-driving trolley.

Re: Ask HN: Is “prompt injection” going to be a new common vulnerability?

#46

Post by franga2000 »

CGamesPlay wrote: post #19

The string-based content moderation is also a laughably cheap hack put in to cover the PR pieces. ChatGPT speaks most human languages, but the content filters only apply in English! The ethics training they did with the model does apply to other languages, indicating that this is a much better avenue for getting outputs you like. But is this a "vulnerability"? No. Presently the only thing these systems can do is "acc…

Returning a string can be plenty dangerous if that string is used somewhere it shouldn't be. A great demo of this was a challenge at DiceCTF [0] where a model was used to generate a string containing placeholders, which was then fed into Python's str.format() function. You could trivially "trick" the model into outputting whatever you wanted and, due to some useful but dangerous Python features, could use the f-string to dump the environment variables (which was the objective here, but you could just as easily access other information in memory).

[0] https://ctftime.org/task/24223

Re: Ask HN: Is “prompt injection” going to be a new common vulnerability?

#47

Post by blackflame7000 »

luckylion wrote: post #44
simonw wrote: post #41

Earlier quoted context omitted.

There was a report just a few days ago of a system that was passing output to the Python eval() function - someone used that to steal an OpenAI API key: https://twitter.com/ludwig_stumpp/status/1619701277419794435 It's vitally important that anyone building against language models like GPT3 understands prompt injection in depth, so they don't make mistakes like this.

API response => eval() it's going to be an interesting future.

[dead]

Re: Ask HN: Is “prompt injection” going to be a new common vulnerability?

#48

Post by chowells »

nhchris wrote: post #45
andrewmcwatters wrote: post #3

If I understand correctly, ChatGPT doesn't have its latent capabilities removed. Instead, they're suppressed by training using negative feedback. These special prompts are supposed to find the remaining stochastic spaces where ChatGPT can process the desired output that is not suppressed by training. So, the danger seems to be that there is no currently documented way to completely remove these possible outputs, beca…

> the danger seems to be that there is no currently documented way to completely remove these possible outputs I know OpenAI likes throwing around terms like "danger" and "harm" liberally, but is this really a danger ? Outside of hypothetical scenarios where someone wires ChatGPT to a self-driving trolley.

Yes, there are absolutely dangers. It shouldn't be possible for a depressed person to convince a chatbot to tell them to commit suicide. There are some people who only need the tiniest push on a bad day.

Re: Ask HN: Is “prompt injection” going to be a new common vulnerability?

#49

Post by bawolff »

A vulnerability? Yes. A serious one... i don't really think so in the grand scheme of things.

Injection vulnerabilities in one form or another are like 90% of all security vulnerabilities. We have the obvious ones like sql injection or shell injection. We dont call XSS injection but it really is just html/js injection. Even things like buffer overflows are injections if viewed through the right lens.

If there is one thing the security field has learned from all this, its that blacklist approaches to security are a pain and almost never work. Especially for complex input formats.

Re: Ask HN: Is “prompt injection” going to be a new common vulnerability?

#50

Post by nhchris »

chowells wrote: post #48
nhchris wrote: post #45

Earlier quoted context omitted.

> the danger seems to be that there is no currently documented way to completely remove these possible outputs I know OpenAI likes throwing around terms like "danger" and "harm" liberally, but is this really a danger ? Outside of hypothetical scenarios where someone wires ChatGPT to a self-driving trolley.

Yes, there are absolutely dangers. It shouldn't be possible for a depressed person to convince a chatbot to tell them to commit suicide. There are some people who only need the tiniest push on a bad day.

By that logic they shouldn't visit the Grand Canyon either, because their echo might say something mean. It's the logic of banning sad songs from the radio. "Tiniest pushes" are omnipresent, calling them "dangerous" stretches that word to meaninglessness.
Post reply on HN