Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Prompt injection explained, with video, slides, and a transcript

simonwillison.net

Post reply on HN
Print view View original

151–160 of 183 posts

Re: Prompt injection explained, with video, slides, and a transcript

#151

Post by simonw »

danShumway wrote: post #149
byteknight wrote: post #147

Earlier quoted context omitted.

SQL injection enters the chat

I'm a little cautious of comparisons to SQL injection now, because while some of the comparisons are very valid (particularly around the risks), prompt injection isn't really the same category of vulnerability as SQL injection -- so mitigation techniques for SQL injection (escaping input, sanitizing) aren't going to work to stop prompt injection. But otherwise yeah, it can be helpful to think of prompt injection as i…

Yeah, the confusion between jailbreaking and prompt injection is definitely a big problem.

People who are frustrated at the safety measure that jailbreaking aims to defeat often assume prompt injection is equally "harmless" - they fail to understands that the consequences can be a lot more severe to anyone who is trying to build their own software on top of LLMs.

Re: Prompt injection explained, with video, slides, and a transcript

#152

Post by simonw »

danShumway wrote: post #148
permo-w wrote: post #139

Earlier quoted context omitted.

>I suspect it'll be a little bit tricky with some applications to keep track of what data is "infected" and what data isn't and when it's appropriate to allow that infected data to be mixed together even with itself could you give an example of an application like this? >extracting a label from the untrusted LLM I concur, you’d have to be very careful with how you generate filenames and metadata. let’s say our system…

In a sibling comment I theorize about how an email summarizer could fall foul of this: ---- As an example, let's say you're coding this up and you decide that for summaries, your sandboxed AI gets all of the messages together in one pass. That would be both cheaper and faster to run and simpler architecture, right? Except it opens you up to a vulnerability, because now an email can change the summary of a different e…

That is a really excellent explanation of why even summarizing trusted and untrusted messages together can cause big problems.

Re: Prompt injection explained, with video, slides, and a transcript

#153

Post by simonw »

danShumway wrote: post #146
cubefox wrote: post #142

Earlier quoted context omitted.

> Quite honestly, that makes me less likely to believe your solution will work. Are you training an LLM to only obey instructions within a given context, or are you training it to recognize prompt injection and avoid it? The former. During instruction tuning, the model learns to "predict" text as if the document describes a dialogue. We then just add examples where special quotes are present, including examples where…

> It's like a browser. You can't prove that Firefox has no security flaws. I've seen this comparison come up a few times and I feel like it's really stretching tbh. Imagine if someone came out with an encryption algorithm, and somebody asked, "okay, but do we know that this is secure" and they said "how do we know anything is secure?" -- what would your response to that person be? And sure, I don't know that Firefox…

Thanks for this - you're making really excellent arguments here.

Re: Prompt injection explained, with video, slides, and a transcript

#154

Post by simonw »

charrondev wrote: post #140
danShumway wrote: post #65

Earlier quoted context omitted.

There's a couple of different stages people tend to go through when learning about prompt injection: A) this would only allow me to break my own stuff, so what's the risk? I just won't break my own stuff. B) surely that's solveable with prompt engineering. C) surely that's solveable with reinforcement training, or chaining LLMs, or . D) okay, but even so, it's not like people are actually putting LLMs into applicatio…

Honestly the it seems like they play for wiring up an LLM to something can actually take action is to only give the LLM the same access that the same user querying your API would have. I’ve been exploring an LLM -> API layer for our app and I’m not worried about prompt Injection because if the user was actually malicious they could just used the interface or the API to do the same thing. In other words if you treat t…

If you're completely confident that there's no way an attacker might get their text into your user's LLM session then yeah, you have nothing to worry about.

Potential vectors to consider:

- Your app lets users run it against text from other sources - fetched web pages, incoming messages - server logs - which an attacker might be able to influence

- Your users can copy and paste text into your app - and an attacker might be able to trick them into eg copying in a dozen paragraphs of text without first reading it to check for weird hidden prompt instructions

Re: Prompt injection explained, with video, slides, and a transcript

#155

Post by jimbokun »

callalex wrote: post #115
theLiminator wrote: post #80

Earlier quoted context omitted.

And then gdpr fucks that up that nice clean concept completely

If it’s so hard to be a good steward of data, don’t collect it in the first place.

It’s not that GDPR is overly onerous to implement. It’s simply that GDPR is fundamentally incompatible with unlimited undo.

Re: Prompt injection explained, with video, slides, and a transcript

#156

Post by jimbokun »

pyth0 wrote: post #138
thornewolf wrote: post #129

Earlier quoted context omitted.

"It merely has all of the byproducts of intelligence, its not intelligence though!" I make this statement in a frank way to rhetorically get the point across. I find myself continually surprised by the general community's desire to reject the intelligence claim in its entirely. I make no claim that this intelligence manifest in the same way human intelligence does. I make no claim that this intelligence can even be m…

> Intelligence is best measured by outcomes. How some entity is best able to manipulate its existence. I agree, although I think we humans have always been fairly bad at measuring intelligence in a way that truly appreciates all the complexity of it. The second part of that is also interesting and I would agree that is partly what makes these LLMs non-intelligent. The models do not really have "an existence" outside…

You can define “intelligence” to only refer to biological intelligence. But that doesn’t mean that AIs can’t do things we call intelligent in humans at or beyond a human level.

Re: Prompt injection explained, with video, slides, and a transcript

#159

Post by scott_w »

thornewolf wrote: post #129
pyth0 wrote: post #114

Earlier quoted context omitted.

The amount of anthropomorphizing of these LLMs in this thread is off the charts. These language models do not have human intelligence, nor do they approximate it, though they do an incredible job at mimicking what the result of intelligence looks like. They are susceptible to prompt injection precisely because of this, and it is why I don't know if it can ever be 100% solved with these models.

"It merely has all of the byproducts of intelligence, its not intelligence though!" I make this statement in a frank way to rhetorically get the point across. I find myself continually surprised by the general community's desire to reject the intelligence claim in its entirely. I make no claim that this intelligence manifest in the same way human intelligence does. I make no claim that this intelligence can even be m…

Because while it’s mimicking a human kind of intelligence, it’s missing the kinds of intelligence that even basic mammals have.

One example: it has no concept of objects and permanence. Something even my dog has.

Want an example? Watch Gotham Chess on YouTube play it at chess and you’ll see it not only doesn’t understand the rules of the game, it can’t even remember which pieces are on the board!

Re: Prompt injection explained, with video, slides, and a transcript

#160

Post by spullara »

wll wrote: post #42
spullara wrote: post #35

Earlier quoted context omitted.

I don't think SnapChat's LLM has access to your location. I think a service that it uses has access to your location and it can't get it directly but it can ask for "restaurants nearby".

Here’s the full Snapchat MyAI prompt. The location is inserted into the system message. Look at the top right. [0] [1] Snapchat asks for the location permission through native APIs or obviously geolocates the user via IP. Either way, it’s fascinating that: people don’t expect it to know their location; don’t expect it to lie; the model goes against its own rules and ”forgets” and “gaslights.” [0] https://www.reddit.c…

Proven wrong thanks. But there is no reason for it to have access and doing it the way I suggested they already were is superior :)
Post reply on HN