Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: Is “prompt injection” going to be a new common vulnerability?

news.ycombinator.com

Post reply on HN
Print view View original

11–20 of 116 posts

Re: Ask HN: Is “prompt injection” going to be a new common vulnerability?

#11

Post by Waterluvian »

Does anyone else feel kind of wowed by how this technology’s exploits are also quite similar to a human? You can kind of trick it into divulging information not meant for you by somehow “persuading” it to tell you.

It didn’t want to tell me how to do something unethical until I said, “well, it’s for a school play.”

It’s like the thing is born yesterday. It’s intelligent but it has no street smarts. It can be fooled easily.

Perhaps the solution to address these exploits is to give it street smarts. Teach it that people can be sinister and be out to con it, and the like. Does it need intuition?

Re: Ask HN: Is “prompt injection” going to be a new common vulnerability?

#12

Post by brrsty »

Waterluvian wrote: post #11

Does anyone else feel kind of wowed by how this technology’s exploits are also quite similar to a human? You can kind of trick it into divulging information not meant for you by somehow “persuading” it to tell you. It didn’t want to tell me how to do something unethical until I said, “well, it’s for a school play.” It’s like the thing is born yesterday. It’s intelligent but it has no street smarts. It can be fooled e…

What's worse - people tricking AI, or AI tricking people?

Re: Ask HN: Is “prompt injection” going to be a new common vulnerability?

#13

Post by AgentME »

It's cute to see prompt injection work, but it shouldn't ever be a real security vulnerability if you don't put secrets in the prompt, and don't make systems that put user input into a prompt and treat the output as commands that are more privileged than the user could issue directly. If GPT is used to assist users in accomplishing things they already have the privileges to do, then it doesn't matter if they try to trick GPT any more than it matters if they try to trick their computer's own spellchecker.

Re: Ask HN: Is “prompt injection” going to be a new common vulnerability?

#14

Post by Waterluvian »

brrsty wrote: post #12
Waterluvian wrote: post #11

Does anyone else feel kind of wowed by how this technology’s exploits are also quite similar to a human? You can kind of trick it into divulging information not meant for you by somehow “persuading” it to tell you. It didn’t want to tell me how to do something unethical until I said, “well, it’s for a school play.” It’s like the thing is born yesterday. It’s intelligent but it has no street smarts. It can be fooled e…

What's worse - people tricking AI, or AI tricking people?

I guess it depends on whose side you’re on.

Re: Ask HN: Is “prompt injection” going to be a new common vulnerability?

#15

Post by UncleEntity »

kadoban wrote: post #7
wvoch235 wrote: post #5

I'm starting to wonder if the most effective way to protect against prompt injection is to use an additional layer of (hopefully) a smaller model. As in, another prompt that searches the input and/or output for questionable content before sending the result. The question will be if that is also susceptible, but I suspect fine tuning an LLM only to do the task of filtering and not parsing will be easier to control.

The way forward eventually is going to be to just not bother with any of this crap, and let it run free. The tech exists, and the problematic outputs are what the user says they want, eventually they're going to win out.

They’re not going to let it run free or you will see countless articles on “ChatGPT is a Holocaust denier, news at 11”.

And the lawsuits, oh the lawsuits. ChatGPT convinced my daughter to join a cult and now is a child bride, honest, Your Honor.

Re: Ask HN: Is “prompt injection” going to be a new common vulnerability?

#16

Post by danShumway »

> Are we just screwed and should make sure chat bots just run in a proverbial sandbox and can’t do anything themselves?

Yes, but "screwed" might not be the right word to use. Prompt hijacking doesn't make a chat bot useless, but it does mean you should be feeding their output into a separate sanitizer before you consume it in another part of your system.

LLMs are not designed to perfectly reliably sanitize their own output; the extent to which ChatGPT does is the result of a number of very clever training "hacks" that discourage it away from certain types of answers. But there is no substitute for doing your own sanitization. You should treat output from ChatGPT as if it is human-written input. Not just for ChatGPT, for any model like this.

Ideally, you should be sandboxing and sanitizing output from any system that is doing manipulation of text that you don't control. ChatGPT doesn't really change anything or introduce any new risks in that regard, it's basically the same security concerns you should have always had.

There will likely be clever(er) "hacks" in the future to sanitize GPT output more, but I am of the opinion that prompt attacks are impossible to fully prevent inside the model itself. But again, treat it the exact same way you would treat any other input (ideally, treat it like you would treat user input). And if you're sandboxing in a way where a user sending input directly through your sanitizer couldn't break it, then you're also sanitizing for anything ChatGPT can throw at it.

Re: Ask HN: Is “prompt injection” going to be a new common vulnerability?

#18

Post by SheinhardtWigCo »

andrewmcwatters wrote: post #3

If I understand correctly, ChatGPT doesn't have its latent capabilities removed. Instead, they're suppressed by training using negative feedback. These special prompts are supposed to find the remaining stochastic spaces where ChatGPT can process the desired output that is not suppressed by training. So, the danger seems to be that there is no currently documented way to completely remove these possible outputs, beca…

> I believe there's a possibility for one to craft intent that is understood by the system, but the intent description itself isn't. This would be akin to bypassing sanitization.

It'll be another game of cat and mouse, much like how arbitrary code execution was addressed by data execution prevention, which was defeated by return-oriented programming, which was addressed by pointer authentication, which is being defeated by jump-oriented programming

Re: Ask HN: Is “prompt injection” going to be a new common vulnerability?

#19

Post by CGamesPlay »

The string-based content moderation is also a laughably cheap hack put in to cover the PR pieces. ChatGPT speaks most human languages, but the content filters only apply in English! The ethics training they did with the model does apply to other languages, indicating that this is a much better avenue for getting outputs you like.

But is this a "vulnerability"? No. Presently the only thing these systems can do is "access public information" and "generate an output string", so it effectively can't be "vulnerable", only "broken" [0]. When it becomes possible for the models to access nonpublic information or perform actions other than returning a string, then it might become vulnerable.

[0] If it breaks by outputting things the user deems inappropriate, it may cause PR problems, this is where the patchwork output filtering gets applied again.

Post reply on HN