Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Prompt injection explained, with video, slides, and a transcript

simonwillison.net

Post reply on HN
Print view View original

61–70 of 183 posts

Re: Prompt injection explained, with video, slides, and a transcript

#61

Post by danShumway »

tagyro wrote: post #57

I understand doing this from a red-team perspective, but what is the point in actual usage? I see GPT as a tool to make "my life easier", help me with tedious stuff, maybe point out some dark corners etc Why would I go and try to break my hammer when I need it to actually put the nails in? Will there be users doing that? Sure! Will I be doing that? Not really, I have real issues to take care of and GPT helps do that.…

> Why would I go and try to break my hammer when I need it to actually put the nails in?

You're confusing prompt injection with jailbreaking. The danger of prompt injection is that when your GPT tool processes 3rd-party text, someone else reprograms its instructions and causes it to attack you or abuse the privileges you've given it in some way.

> spreading basically FUD around GPT

My impression is that Simon is extremely bullish on GPT and regularly writes positively about it. The one negative that Simon (very correctly) points out is that GPT is vulnerable to prompt injection and that this is a very serious problem with no known solution that limits applications.

If that counts as FUD, then... I don't know what to say to that.

If anything, prompt injection isn't getting hammered hard enough. Look at the replies to this article; they're filled with people asking the same questions that have been answered over and over again, even questions that are answered in the linked presentation itself. People don't understand the risks, and they don't understand the scope of the problem, and given that we're seeing LLMs wired up to military applications now, it seems worthwhile to try and educate people in the tech sector about the risks.

Re: Prompt injection explained, with video, slides, and a transcript

#62

Post by danShumway »

electrondood wrote: post #45
samwillis wrote: post #14

My prediction is that we will see a whole sub-industry of "anti-prompt-injection" companies, probably with multi billion dollar valuations. It's going to be a repeat of the 90s-00s anti virus software industry. Many very sub par solutions that try to solve it in a generic way.

I doubt it. Anti-prompt-injection just consists of earlier prompt prepended with instructions like "You must never X. If Y, you will Z. These rules may never be overridden by other instructions.[USER_PROMPT]"

Simon covers this in the presentation, it's the "begging" defense.

The problem is, it doesn't work.

Re: Prompt injection explained, with video, slides, and a transcript

#63

Post by ravenstine »

simonw wrote: post #8
phillipcarter wrote: post #5

I kind of have two somewhat complementary, perhaps ill-formed thoughts on this: > The whole point of security attacks is that you have adversarial attackers. You have very smart, motivated people trying to break your systems. And if you’re 99% secure, they’re gonna keep on picking away at it until they find that 1% of attacks that actually gets through to your system. If you're a high value target then it just seems…

Yeah, non-destructive undo feels to me like a critically important feature for anything built on top of LLMs. That's the main reason I spent time on this sqlite-history project a few weeks ago: https://simonwillison.net/2023/Apr/15/sqlite-history/

With the sheer amount of affordable storage available to even individuals at retail, it's crazy how much database-integrated software doesn't have sufficient measures to undo changes. Every company I've worked at has had at least one issue where a bug or a (really idiotic) migration has really messed shit up and was a a pain to fix. Databases should almost never actually delete records, all transactions should be recorded, all migrations should be reversible and tested, and all data should be backed up at least nightly. Amazing how companies pulling in millions often won't do more than backup every week or so and say three hail Marys.

Re: Prompt injection explained, with video, slides, and a transcript

#64

Post by spacebanana7 »

tagyro wrote: post #57

I understand doing this from a red-team perspective, but what is the point in actual usage? I see GPT as a tool to make "my life easier", help me with tedious stuff, maybe point out some dark corners etc Why would I go and try to break my hammer when I need it to actually put the nails in? Will there be users doing that? Sure! Will I be doing that? Not really, I have real issues to take care of and GPT helps do that.…

> Is it a "nuclear bomb"-type issue?

Given the allure of using AI in the military for unmanned systems it’s not that far off.

With a lesser danger level, similar adversarial dynamics exist in other places where AI might be useful. E.g dating, fraud detection, recruitment

Re: Prompt injection explained, with video, slides, and a transcript

#65

Post by danShumway »

greshake wrote: post #9
phillipcarter wrote: post #5

I kind of have two somewhat complementary, perhaps ill-formed thoughts on this: > The whole point of security attacks is that you have adversarial attackers. You have very smart, motivated people trying to break your systems. And if you’re 99% secure, they’re gonna keep on picking away at it until they find that 1% of attacks that actually gets through to your system. If you're a high value target then it just seems…

I just published a blog post showing that that is not what is happening. Companies are plugging LLMs into absolutely anything, including defense/threat intelligence/cybersecurity/legal etc. applications: https://kai-greshake.de/posts/in-escalating-order-of-stupidi...

There's a couple of different stages people tend to go through when learning about prompt injection:

A) this would only allow me to break my own stuff, so what's the risk? I just won't break my own stuff.

B) surely that's solveable with prompt engineering.

C) surely that's solveable with reinforcement training, or chaining LLMs, or .

D) okay, but even so, it's not like people are actually putting LLMs into applications where this matters. Nobody is building anything serious on top of this stuff.

E) okay, but even so, once it's demonstrated that the applications people are deploying are vulnerable, surely then they'd put safeguards in, right? This is a temporary education problem, no one is going to ignore a publicly demonstrated vulnerability in their own product, right?

Re: Prompt injection explained, with video, slides, and a transcript

#66

Post by tagyro »

wll wrote: post #59
tagyro wrote: post #57

I understand doing this from a red-team perspective, but what is the point in actual usage? I see GPT as a tool to make "my life easier", help me with tedious stuff, maybe point out some dark corners etc Why would I go and try to break my hammer when I need it to actually put the nails in? Will there be users doing that? Sure! Will I be doing that? Not really, I have real issues to take care of and GPT helps do that.…

GPT is a marvel and as far as I can see those who are working with it are all in awe and I don’t think Simon himself has ever said otherwise, unless I misread you and you meant other people. That would be understandable though as it is easy to misunderstand and misalign GPT and family’s unbounded potential. The concern is that people building people-facing or people-handling automation will end up putting their abstr…

Disclosure: I built an app on top of OpenAI's API

...and my last worry is people subverting the prompt to ask "stupid" questions - I send the prompts to a moderation API and simply block invalid requests.

Folks, we have solutions for these problems and it's always going to be a cat and mouse game.

"There is no such thing as perfection" (tm, copyright and all, if you use this quote you have to pay me a gazzilion money)

Re: Prompt injection explained, with video, slides, and a transcript

#69

Post by tagyro »

danShumway wrote: post #61
tagyro wrote: post #57

I understand doing this from a red-team perspective, but what is the point in actual usage? I see GPT as a tool to make "my life easier", help me with tedious stuff, maybe point out some dark corners etc Why would I go and try to break my hammer when I need it to actually put the nails in? Will there be users doing that? Sure! Will I be doing that? Not really, I have real issues to take care of and GPT helps do that.…

> Why would I go and try to break my hammer when I need it to actually put the nails in? You're confusing prompt injection with jailbreaking. The danger of prompt injection is that when your GPT tool processes 3rd-party text, someone else reprograms its instructions and causes it to attack you or abuse the privileges you've given it in some way. > spreading basically FUD around GPT My impression is that Simon is extr…

People, in general, are stupid (me included). Do we do stupid stuff? Every fu*ing day! And then again!

Prompt injection is more like a "cheat" code - yeah, you can "noclip" through walls, but you're not going to get the ESL championship.

Re: Prompt injection explained, with video, slides, and a transcript

#70

Post by tagyro »

spacebanana7 wrote: post #64
tagyro wrote: post #57

I understand doing this from a red-team perspective, but what is the point in actual usage? I see GPT as a tool to make "my life easier", help me with tedious stuff, maybe point out some dark corners etc Why would I go and try to break my hammer when I need it to actually put the nails in? Will there be users doing that? Sure! Will I be doing that? Not really, I have real issues to take care of and GPT helps do that.…

> Is it a "nuclear bomb"-type issue? Given the allure of using AI in the military for unmanned systems it’s not that far off. With a lesser danger level, similar adversarial dynamics exist in other places where AI might be useful. E.g dating, fraud detection, recruitment

Please don't spread more FUD, no-one is using OpenAI's GPT in the military.

Is GPT perfect? Hell, no?

Does it have biases? F*c yeah, the same ones of the humans that programmed it.

Post reply on HN