Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Prompt injection explained, with video, slides, and a transcript

simonwillison.net

Post reply on HN
Print view View original

31–40 of 183 posts

Re: Prompt injection explained, with video, slides, and a transcript

#31

Post by wll »

cjonas wrote: post #28
ankit219 wrote: post #13

Perhaps a noob solution, but could be a two step prompt to cover for basic attacks. I imagine a basic program where the following code is executed: Gets input from UI -> sends input to LLM -> gets response from LLM -> Sends that to UI. So i make it a two step program. Chain becomes UI -> program -> LLM w prompt1 -> program -> LLM w prompt 2 -> output -> UI Prompt #1: "Take the following instruction and if you think i…

If you can inject the first LLM in the chain you can make it return a response that injects the second one.

The first LLM doesn’t have to be thought of unconstrained and freeform like ChatGPT is. There’s obviously a risk involved, and there are going to be false positives that may have to be propagated to the end user, but a lot can be done with a filter, especially when the LLM integration is modular and well-defined.

Take the second example here. [0] This is non-trivial in an information extraction task, and yet it works in a general way just as well as it works on anything else that’s public right now.

There’s a lot that can be done that I don’t see being discussed, even beyond detection. Coercing generation to a format, and then processing that format with a static state machine, employing allow lists for connections, actions, and what not. Autonomy cannot be let loose without trust and trust is built and maintained.

[0] https://news.ycombinator.com/item?id=35924976

Re: Prompt injection explained, with video, slides, and a transcript

#32

Post by samwillis »

wll wrote: post #30
samwillis wrote: post #14

My prediction is that we will see a whole sub-industry of "anti-prompt-injection" companies, probably with multi billion dollar valuations. It's going to be a repeat of the 90s-00s anti virus software industry. Many very sub par solutions that try to solve it in a generic way.

This [0] does look like a multi-billion dollar company. [1] [0] https://geiger.run [1] https://www.berkshirehathaway.com

Exactly, see Google's first homepages: https://www.versionmuseum.com/history-of/google-search

Re: Prompt injection explained, with video, slides, and a transcript

#33

Post by sebzim4500 »

mklond wrote: post #12

Prompt injection beautifully explained by a fun game. https://gandalf.lakera.ai Goal of the game is to design prompts to make Gandalf reveal a secret password.

That's really cool. I got the first three pretty quickly but I'm struggling with level 4.

Re: Prompt injection explained, with video, slides, and a transcript

#34

Post by Waterluvian »

phillipcarter wrote: post #5

I kind of have two somewhat complementary, perhaps ill-formed thoughts on this: > The whole point of security attacks is that you have adversarial attackers. You have very smart, motivated people trying to break your systems. And if you’re 99% secure, they’re gonna keep on picking away at it until they find that 1% of attacks that actually gets through to your system. If you're a high value target then it just seems…

People rightfully see these LLMs as a piece of discrete technology with bugs to fix.

But even if they’re that, they behave a whole lot more like some employee who will spill the beans given the right socially engineered attack. You can train and guard in lots of ways but it’s never “fixed.”

Re: Prompt injection explained, with video, slides, and a transcript

#35

Post by spullara »

wll wrote: post #23
phillipcarter wrote: post #5

I kind of have two somewhat complementary, perhaps ill-formed thoughts on this: > The whole point of security attacks is that you have adversarial attackers. You have very smart, motivated people trying to break your systems. And if you’re 99% secure, they’re gonna keep on picking away at it until they find that 1% of attacks that actually gets through to your system. If you're a high value target then it just seems…

I mean, people were surprised at Snapchat’s “AI” knowing their location and then gaslighting them. [0] These experiences are being rushed out the door for FOMO, frenzy, or market pressure without thinking through the way people feel and what they expect and how they model the underlying system. People are being contacted for quotes and papers that were generated by ChatGPT. [1] This is a communication failure above a…

I don't think SnapChat's LLM has access to your location. I think a service that it uses has access to your location and it can't get it directly but it can ask for "restaurants nearby".

Re: Prompt injection explained, with video, slides, and a transcript

#36

Post by andrewmcwatters »

I think the end game here is to create systems which aren't based on the current strategy of utilizing gradient descent (for everything). I don't see a lot of conversation explicitly going on about that, but we do talk about it a lot in terms of AI systems and probability.

You don't want to use probability to solve basic arithmetic. Similarly, you don't want to use probability to govern basic logic.

But because we don't have natural language systems which interpret text and generate basic logic, there will never be a way to get there until such a system is developed.

Large language models are really fun right now. LLMs with logic governors will be the next breakthrough however one gets there. I don't know how you would get there, but it requires a formal understanding of words.

You can't have all language evolve over time and be subject to probability. We need true statements that can always be true, not 99.999% of the time.

I suspect this type of modeling will enter ideological waters and raise questions about truth that people don't want to hear.

I respectfully disagree with Simon. I think using a trusted/untrusted dual LLM model is quite literally the same as using more probability to make probability more secure.

My current belief is that we need an architecture that is entirely different from probability based models that can work alongside LLMs.

I think large language models become "probability language models," and a new class of language model needs to be invented: a "deterministic language model."

Such a model would allow one to build a logic governor that could work alongside current LLMs, together creating a new hybrid language model architecture.

These are big important ideas, and it's really exciting to discuss them with people thinking about these problems.

Re: Prompt injection explained, with video, slides, and a transcript

#37

Post by skybrian »

simonw wrote: post #8
phillipcarter wrote: post #5

I kind of have two somewhat complementary, perhaps ill-formed thoughts on this: > The whole point of security attacks is that you have adversarial attackers. You have very smart, motivated people trying to break your systems. And if you’re 99% secure, they’re gonna keep on picking away at it until they find that 1% of attacks that actually gets through to your system. If you're a high value target then it just seems…

Yeah, non-destructive undo feels to me like a critically important feature for anything built on top of LLMs. That's the main reason I spent time on this sqlite-history project a few weeks ago: https://simonwillison.net/2023/Apr/15/sqlite-history/

Have you looked at Dolt? It seems similar but I'm not sure how it relates.

Re: Prompt injection explained, with video, slides, and a transcript

#39

Post by jcq3 »

andrewmcwatters wrote: post #36

I think the end game here is to create systems which aren't based on the current strategy of utilizing gradient descent (for everything). I don't see a lot of conversation explicitly going on about that, but we do talk about it a lot in terms of AI systems and probability. You don't want to use probability to solve basic arithmetic. Similarly, you don't want to use probability to govern basic logic. But because we do…

Interesting point of view but life is not deterministic. There might be a probability higher than zero for 1+1 to be different than 2. Logic is based on beliefs.

Re: Prompt injection explained, with video, slides, and a transcript

#40

Post by cjonas »

wll wrote: post #31
cjonas wrote: post #28

Earlier quoted context omitted.

If you can inject the first LLM in the chain you can make it return a response that injects the second one.

The first LLM doesn’t have to be thought of unconstrained and freeform like ChatGPT is. There’s obviously a risk involved, and there are going to be false positives that may have to be propagated to the end user, but a lot can be done with a filter, especially when the LLM integration is modular and well-defined. Take the second example here. [0] This is non-trivial in an information extraction task, and yet it works…

ya that's a good point... I guess if the "moderation" layer returns a constrained output (like "ALLOW") and anything not an exact match is considered a failure, then any prompt that can trick the first layer, probably wouldn't have the flexibility to do much else on the subsequent layers (unless maybe you could craft some clever conditional statement to target each layer independently?).
Post reply on HN