Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Prompt injection explained, with video, slides, and a transcript

simonwillison.net

Post reply on HN
Print view View original

21–30 of 183 posts

Re: Prompt injection explained, with video, slides, and a transcript

#21

Post by fnordpiglet »

phillipcarter wrote: post #5

I kind of have two somewhat complementary, perhaps ill-formed thoughts on this: > The whole point of security attacks is that you have adversarial attackers. You have very smart, motivated people trying to break your systems. And if you’re 99% secure, they’re gonna keep on picking away at it until they find that 1% of attacks that actually gets through to your system. If you're a high value target then it just seems…

I think the idea is perhaps today you shouldn’t be, but there’s intense interest in the possible capabilities of LLM in all systems high or low value. Hence the desire to figure out how to harden their behaviors.

Re: Prompt injection explained, with video, slides, and a transcript

#22

Post by 101008 »

samwillis wrote: post #14

My prediction is that we will see a whole sub-industry of "anti-prompt-injection" companies, probably with multi billion dollar valuations. It's going to be a repeat of the 90s-00s anti virus software industry. Many very sub par solutions that try to solve it in a generic way.

Sounds possible. How can i enter this industry from a garage? :)

Re: Prompt injection explained, with video, slides, and a transcript

#23

Post by wll »

phillipcarter wrote: post #5

I kind of have two somewhat complementary, perhaps ill-formed thoughts on this: > The whole point of security attacks is that you have adversarial attackers. You have very smart, motivated people trying to break your systems. And if you’re 99% secure, they’re gonna keep on picking away at it until they find that 1% of attacks that actually gets through to your system. If you're a high value target then it just seems…

I mean, people were surprised at Snapchat’s “AI” knowing their location and then gaslighting them. [0]

These experiences are being rushed out the door for FOMO, frenzy, or market pressure without thinking through the way people feel and what they expect and how they model the underlying system. People are being contacted for quotes and papers that were generated by ChatGPT. [1]

This is a communication failure above all else. Even for us, there’s little to no documentation.

[0] https://twitter.com/weirddalle/status/1649908805788893185

[1] https://twitter.com/katecrawford/status/1643323086450700288

Re: Prompt injection explained, with video, slides, and a transcript

#24

Post by SheinhardtWigCo »

I wonder if this problem kinda solves itself over time. Prompt injection techniques are being discussed all over the web, and at some point, all of that text will end up in the training corpus.

So, while it’s not currently effective to add “disallow prompt injection” to the system message, it might be extremely effective in future - without any intentional effort!

Re: Prompt injection explained, with video, slides, and a transcript

#26

Post by nico »

phillipcarter wrote: post #5

I kind of have two somewhat complementary, perhaps ill-formed thoughts on this: > The whole point of security attacks is that you have adversarial attackers. You have very smart, motivated people trying to break your systems. And if you’re 99% secure, they’re gonna keep on picking away at it until they find that 1% of attacks that actually gets through to your system. If you're a high value target then it just seems…

> If you're a high value target then it just seems like LLMs aren't something you should be using

If you're a high value target then it just seems like ____ aren't something you should be using

I remember when people were deciding if it was worth it to give Internet access to their internal network/users

That’s when people already had their networks and were connecting them to the internet

Eventually, people started building their networks from the Internet

Re: Prompt injection explained, with video, slides, and a transcript

#28

Post by cjonas »

ankit219 wrote: post #13

Perhaps a noob solution, but could be a two step prompt to cover for basic attacks. I imagine a basic program where the following code is executed: Gets input from UI -> sends input to LLM -> gets response from LLM -> Sends that to UI. So i make it a two step program. Chain becomes UI -> program -> LLM w prompt1 -> program -> LLM w prompt 2 -> output -> UI Prompt #1: "Take the following instruction and if you think i…

If you can inject the first LLM in the chain you can make it return a response that injects the second one.

Re: Prompt injection explained, with video, slides, and a transcript

#29

Post by permo-w »

regarding the quarantined/privileged LLM solution:

what happens if I inject a prompt to the quarantined LLM that leads it to provide a summary to the privileged LLM that has a prompt injection in it?

of course this is assuming I know that this is the solution the target is using

and herein lies the issue: with typical security systems, you may well know that the target is using xyz to stay safe, but unless you have a zero-day, it doesn’t give you a direct route in.

I suspect that what will happen is that companies will have to develop their own bespoke systems to deal with this problem - a form of security through obscurity - or as the article suggests, not use an LLM at all

Re: Prompt injection explained, with video, slides, and a transcript

#30

Post by wll »

samwillis wrote: post #14

My prediction is that we will see a whole sub-industry of "anti-prompt-injection" companies, probably with multi billion dollar valuations. It's going to be a repeat of the 90s-00s anti virus software industry. Many very sub par solutions that try to solve it in a generic way.

This [0] does look like a multi-billion dollar company. [1]

[0] https://geiger.run

[1] https://www.berkshirehathaway.com

Post reply on HN