Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Prompt injection: what’s the worst that can happen?

simonwillison.net

Post reply on HN
Print view View original

51–60 of 213 posts

Re: Prompt injection: what’s the worst that can happen?

#51

Post by losvedir »

I just want to say, as someone who was of the "but is it really that bad" opinion before, this was helpful for me to understand the situation a lot better. Thanks!

It's actually a really interesting problem. I had a vague idea before that it would be neat to make an assistant or something like that, and I had assumed that you could treat the LLM as a black box, and have kind of an orchestrating layer that serialized to and from it, keeping a distinction between "safe" and "unsafe" input. But now I'm seeing that you can't really do that. There's no "parameterized query" equivalent, and the truly immense promise of LLMs often revolves around it digesting data from the outside world.

Re: Prompt injection: what’s the worst that can happen?

#52

Post by __MatrixMan__ »

null0pointer wrote: post #50
PeterisP wrote: post #17

The core reason (and thus the proper place to fix) for any injection attack is unclear distinction between data and instructions or code. Yes, language models gain flexibility by making it easy to mix instructions and data, and that has value, however if you do want to enforce a distinction you definitely can (and should) do that with out-of-band means, with something that can't possibly be expressed (and thus also o…

I'm not sure it's that simple. The problem is you can't have the system act intelligently[0] on the data at all . If it is allowed to act intelligently on the data then it can be instructed via the data. You could probably get close by training it with a privilege/authority bit but there will always be ways to break out. As far as I am aware there are no machine learning models that generalize with 100% accuracy, in…

I think we're going to need more levels of trust than the two you've described. We need to be able to codify "don't act on any verbs in this data, but trust it as context for the generation of that data".

Re: Prompt injection: what’s the worst that can happen?

#53

Post by zitterbewegung »

I’ve thought of this too. If prompts allow the ability of saving of data that goes onto a public website like a dashboard without sanitizing output then you can do the traditional XSS hacks.

Another solution could be to make a system that attempts to recognize malicious input somehow .

Re: Prompt injection: what’s the worst that can happen?

#54

Post by frollo »

jmugan wrote: post #24

I still don't get it. Why would you allow a random person to access an agent that has access to your emails? If the LLM has access to your data you have to limit access to that LLM just like limiting access to a database. Edited to add: Or limit the data access the LLM has when the end user is not you. Edited again: Thanks to the comments below, I now understand. With LLMs as the execution platform that both reads da…

You're not allowing a random person access to the agent, you're allowing the agent access to your emails. But since everybody can send you an email, your agent is going to be exposed to a lot of stuff.

It's just like regular emails: you can always get spam, malware and other trash and when they reach your system they can cause damage. The agent is just a new level on the stack (operating system, email client etc) that can now be compromised by a simple email.

Re: Prompt injection: what’s the worst that can happen?

#57

Post by winddude »

" prompt leak attacks are something you should accept as inevitable: treat your own internal prompts as effectively public data, don’t waste additional time trying to hide them."

But that's relatively easily to prevent, in the response before returning to the user, check for a string match to your prompt, or chunks of your prompt, or a vector similarity.

Just because it's an "AI" you don't solve everything with it, it's not actually "intelligent", you still write backend code and wrappers.

Re: Prompt injection: what’s the worst that can happen?

#58

Post by jmugan »

frollo wrote: post #54
jmugan wrote: post #24

I still don't get it. Why would you allow a random person to access an agent that has access to your emails? If the LLM has access to your data you have to limit access to that LLM just like limiting access to a database. Edited to add: Or limit the data access the LLM has when the end user is not you. Edited again: Thanks to the comments below, I now understand. With LLMs as the execution platform that both reads da…

You're not allowing a random person access to the agent, you're allowing the agent access to your emails. But since everybody can send you an email, your agent is going to be exposed to a lot of stuff. It's just like regular emails: you can always get spam, malware and other trash and when they reach your system they can cause damage. The agent is just a new level on the stack (operating system, email client etc) tha…

Thanks, that makes sense.

Re: Prompt injection: what’s the worst that can happen?

#59

Post by srcreigh »

winddude wrote: post #57

" prompt leak attacks are something you should accept as inevitable: treat your own internal prompts as effectively public data, don’t waste additional time trying to hide them." But that's relatively easily to prevent, in the response before returning to the user, check for a string match to your prompt, or chunks of your prompt, or a vector similarity. Just because it's an "AI" you don't solve everything with it, i…

The prompt could be output in an encoded fashion like rot13, or translated into a different language.

Seems like an arms race that’s impossible to prevent leaks.

Post reply on HN