Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Prompt injection explained, with video, slides, and a transcript

simonwillison.net

Post reply on HN
Print view View original

131–140 of 183 posts

Re: Prompt injection explained, with video, slides, and a transcript

#131

Post by scarface_74 »

pyth0 wrote: post #114
fzeindl wrote: post #94

Earlier quoted context omitted.

> Prompt injection works because LLMs are dumber than humans at keeping secrets, and humans can be coerced into revealing. I wouldn't say dumber than humans. Actually prompt injections remind me a lot of how you can trick little children into giving up secrets. They are too easily distracted, their thought-structures are free floating and not as fortified as adults. LLMs show childlike intelligence in this regard whi…

The amount of anthropomorphizing of these LLMs in this thread is off the charts. These language models do not have human intelligence, nor do they approximate it, though they do an incredible job at mimicking what the result of intelligence looks like. They are susceptible to prompt injection precisely because of this, and it is why I don't know if it can ever be 100% solved with these models.

This type of comment sounds like the one that comes up anytime someone mentions “serverless”.

“Well there is no such thing as serverless. There are servers in the background”.

Yes people on HN already know that. We also know that Alice and Bob are not real people working in cryptography.

Re: Prompt injection explained, with video, slides, and a transcript

#132

Post by permo-w »

danShumway wrote: post #84
permo-w wrote: post #29

regarding the quarantined/privileged LLM solution: what happens if I inject a prompt to the quarantined LLM that leads it to provide a summary to the privileged LLM that has a prompt injection in it? of course this is assuming I know that this is the solution the target is using and herein lies the issue: with typical security systems, you may well know that the target is using xyz to stay safe, but unless you have a…

> to the quarantined LLM that leads it to provide a summary to the privileged LLM that has a prompt injection in it? In Simon's system, the privileged LLM never gets a summary at all. The quarantined LLM can't talk to it and it can't return any text that the privileged LLM will see. Rather, the privileged LLM executes a function and the text of the quarantined LLM is inserted outside of the LLMs entirely into that fu…

ah I see. thank you for pointing this out

so if I’m reading it correctly now, essentially the quarantined LLM’s outputs are only ever—let’s say—secure text files and the privileged LLM can only ever just point to those text files for the human user to decide what to do with themselves?

I’ll be honest, I quite like how this solution puts a soft cap on how much human interaction automation we can safely get away with, which I think is good in the grand scheme of things

the way I’d implement this would be with a mainloop that iterates over inputs saving each quarantined completion to some form of data storage hardened to classic code injection, then the privileged LLM looks at a carefully curated set of a metadata to decide whether or how to display the results to the user. I suppose there could be some fiddliness in curating the text, and perhaps some level of UI fiddliness in smoothly displaying the completions to user without putting it through the model, but is there more?

Re: Prompt injection explained, with video, slides, and a transcript

#133

Post by robinduckett »

robterrell wrote: post #107
robinduckett wrote: post #103

Can’t you just ask another LLM to analyse the text of the input to determine if it’s an attempted prompt injection?

That's a possible mitigation mentioned in the article.

Maybe I'll read it next time :D

Re: Prompt injection explained, with video, slides, and a transcript

#134

Post by pyth0 »

scarface_74 wrote: post #131
pyth0 wrote: post #114

Earlier quoted context omitted.

The amount of anthropomorphizing of these LLMs in this thread is off the charts. These language models do not have human intelligence, nor do they approximate it, though they do an incredible job at mimicking what the result of intelligence looks like. They are susceptible to prompt injection precisely because of this, and it is why I don't know if it can ever be 100% solved with these models.

This type of comment sounds like the one that comes up anytime someone mentions “serverless”. “Well there is no such thing as serverless. There are servers in the background”. Yes people on HN already know that. We also know that Alice and Bob are not real people working in cryptography.

Yeah no I don't believe this is a fair comparison at all, and I'm frankly surprised you think this is accurate to the discussion around LLMs. There are certainly people on here who believe and talk about ChatGPT as if it is generally intelligent. I suppose if you really want I can look through previous threads, but you really can find this under most threads about ChatGPT. A brand of this fallacious reasoning I find particularly annoying are responses that take the form of "well humans also do " usually in response to people pointing out weaknesses in these language models. It doesn't really matter whether these commenters believe it or not, it does not further the discussion in a meaningful way and it perpetuates FUD around the "AI takeover".

Re: Prompt injection explained, with video, slides, and a transcript

#135

Post by permo-w »

modestygrime wrote: post #123
danShumway wrote: post #84

Earlier quoted context omitted.

> to the quarantined LLM that leads it to provide a summary to the privileged LLM that has a prompt injection in it? In Simon's system, the privileged LLM never gets a summary at all. The quarantined LLM can't talk to it and it can't return any text that the privileged LLM will see. Rather, the privileged LLM executes a function and the text of the quarantined LLM is inserted outside of the LLMs entirely into that fu…

I'm not sure I understand. What is the purpose of the privileged LLM? Couldn't it be replaced with code written by a developer? And aren't you still passing untrusted content into the function call either way? Perhaps a code example of this dual LLM setup would be helpful. Do you know of any examples?

this was my first thought too, but I can see the benefit of it

taking the example from the article, imagine you have a central personal, household or business LLM that you give general verbal or typed commands to and it intelligently converts those commands to system actions.

you say “give a summary of my most recent three emails”, and the power LLM, instead of unsafely going and doing the summaries itself, accesses/generates a quarantined LLM’s summaries, then displays those summaries to you without actually putting the text through its model

I’m building upon the idea here a little, but let’s say you read the summaries and find them trustworthy, you could then say “reply to email 1 in xyz manner” to the privileged power LLM, which then gives a third LLM with email sending privileges access to summary 1’s file

Re: Prompt injection explained, with video, slides, and a transcript

#136

Post by danShumway »

permo-w wrote: post #132
danShumway wrote: post #84

Earlier quoted context omitted.

> to the quarantined LLM that leads it to provide a summary to the privileged LLM that has a prompt injection in it? In Simon's system, the privileged LLM never gets a summary at all. The quarantined LLM can't talk to it and it can't return any text that the privileged LLM will see. Rather, the privileged LLM executes a function and the text of the quarantined LLM is inserted outside of the LLMs entirely into that fu…

ah I see. thank you for pointing this out so if I’m reading it correctly now, essentially the quarantined LLM’s outputs are only ever—let’s say—secure text files and the privileged LLM can only ever just point to those text files for the human user to decide what to do with themselves? I’ll be honest, I quite like how this solution puts a soft cap on how much human interaction automation we can safely get away with,…

> essentially the quarantined LLM’s outputs are only ever—let’s say—secure text files and the privileged LLM can only ever just point to those text files for the human user to decide what to do with themselves?

That's a really good way of putting it. The quarantined outputs are stuck in closed boxes, and the privileged LLM can only ever see the outside of those boxes, not the inside.

> where does the fiddliness come in?

I gave an example in a sibling answer of a common mistake I suspect people would make (having the unprivileged LLM operate on multiple prompts at the same time rather than separately) but it's mostly stuff like that -- I suspect it'll be a little bit tricky with some applications to keep track of what data is "infected" and what data isn't and when it's appropriate to allow that infected data to be mixed together even with itself.

I suspect that for more complicated apps you'll have to be really careful to make sure that there's not some circuitous route where the output of one call gets passed into another one. But it's quite possible I'm overstating the problem. I just worry that someone ends up doing something like extracting a label from the untrusted LLM and sticking into a name or something that the privileged LLM can look at.

Re: Prompt injection explained, with video, slides, and a transcript

#138

Post by pyth0 »

thornewolf wrote: post #129
pyth0 wrote: post #114

Earlier quoted context omitted.

The amount of anthropomorphizing of these LLMs in this thread is off the charts. These language models do not have human intelligence, nor do they approximate it, though they do an incredible job at mimicking what the result of intelligence looks like. They are susceptible to prompt injection precisely because of this, and it is why I don't know if it can ever be 100% solved with these models.

"It merely has all of the byproducts of intelligence, its not intelligence though!" I make this statement in a frank way to rhetorically get the point across. I find myself continually surprised by the general community's desire to reject the intelligence claim in its entirely. I make no claim that this intelligence manifest in the same way human intelligence does. I make no claim that this intelligence can even be m…

> Intelligence is best measured by outcomes. How some entity is best able to manipulate its existence.

I agree, although I think we humans have always been fairly bad at measuring intelligence in a way that truly appreciates all the complexity of it. The second part of that is also interesting and I would agree that is partly what makes these LLMs non-intelligent. The models do not really have "an existence" outside of the moment in which they are processing the context and producing output.

> Intelligence can manifest in more than one way. An entirely mechanical system could be considered to have some level of "intelligence"

I don't think I agree with this, or at least maybe I disagree with your definition of "intelligent". I believe that intelligence is heavily intertwined with biology and it exists is all manner of non-human creatures but I don't think I would call an entirely mechanical system "intelligent". Perhaps I would say it had "intelligent design".

> Considering something intelligent or to have desires is not anthropomorphizing.

I absolutely agree with this and I was not trying to imply that it was unique to humans. In fact I think we severely discount the amount of intelligence in non-human life forms all the time.

I do think that ChatGPT possesses knowledge (as encoded in its weights) similar to a book, however unlike a book it also has a convenient and familiar interface that allows us to interact with this knowledge and form unique and novel results.

Re: Prompt injection explained, with video, slides, and a transcript

#139

Post by permo-w »

danShumway wrote: post #136
permo-w wrote: post #132

Earlier quoted context omitted.

ah I see. thank you for pointing this out so if I’m reading it correctly now, essentially the quarantined LLM’s outputs are only ever—let’s say—secure text files and the privileged LLM can only ever just point to those text files for the human user to decide what to do with themselves? I’ll be honest, I quite like how this solution puts a soft cap on how much human interaction automation we can safely get away with,…

> essentially the quarantined LLM’s outputs are only ever—let’s say—secure text files and the privileged LLM can only ever just point to those text files for the human user to decide what to do with themselves? That's a really good way of putting it. The quarantined outputs are stuck in closed boxes, and the privileged LLM can only ever see the outside of those boxes, not the inside. > where does the fiddliness come…

>I suspect it'll be a little bit tricky with some applications to keep track of what data is "infected" and what data isn't and when it's appropriate to allow that infected data to be mixed together even with itself

could you give an example of an application like this?

>extracting a label from the untrusted LLM

I concur, you’d have to be very careful with how you generate filenames and metadata. let’s say our system does all the things we’ve talked about, but it saves the email sender address plaintext in the meta data. I don’t know the limits on the length of an email, and all the powerful prompt injections I’ve seen are quite long, but there’s an attack surface there, especially if the attacker has knowledge of the system

with regards to names, you’d just have to generate them completely generically, perhaps just with timestamps. anything generated from the actual text would be a massive oversight

Re: Prompt injection explained, with video, slides, and a transcript

#140

Post by charrondev »

danShumway wrote: post #65
greshake wrote: post #9

Earlier quoted context omitted.

I just published a blog post showing that that is not what is happening. Companies are plugging LLMs into absolutely anything, including defense/threat intelligence/cybersecurity/legal etc. applications: https://kai-greshake.de/posts/in-escalating-order-of-stupidi...

There's a couple of different stages people tend to go through when learning about prompt injection: A) this would only allow me to break my own stuff, so what's the risk? I just won't break my own stuff. B) surely that's solveable with prompt engineering. C) surely that's solveable with reinforcement training, or chaining LLMs, or . D) okay, but even so, it's not like people are actually putting LLMs into applicatio…

Honestly the it seems like they play for wiring up an LLM to something can actually take action is to only give the LLM the same access that the same user querying your API would have.

I’ve been exploring an LLM -> API layer for our app and I’m not worried about prompt Injection because if the user was actually malicious they could just used the interface or the API to do the same thing.

In other words if you treat the LLM like any other frontend then you really should have a problem from a security standpoint. Your would have your iOS application super user access your system, why would you treat an LLM different than any other client.

Post reply on HN