Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Prompt injection explained, with video, slides, and a transcript

simonwillison.net

Post reply on HN
Print view View original

81–90 of 183 posts

Re: Prompt injection explained, with video, slides, and a transcript

#81

Post by armchairhacker »

Prompt injection works because LLMs are dumber than humans at keeping secrets, and humans can be coerced into revealing information and doing things they're not supposed to (see: SMS hijacking).

We already have the solution: logical safeguards that make doing the wrong thing impossible, or at least hard. AI shouldn't have access to secret information, it should only have the declassified version (e.g. anonymized statistics, a program which reveals small portions to the AI with a delay); and if users may need to request something more, it should be instructed to connect them to a human agent who is trained on proper disclosure.

Re: Prompt injection explained, with video, slides, and a transcript

#82

Post by quickthrower2 »

armchairhacker wrote: post #81

Prompt injection works because LLMs are dumber than humans at keeping secrets, and humans can be coerced into revealing information and doing things they're not supposed to (see: SMS hijacking). We already have the solution: logical safeguards that make doing the wrong thing impossible, or at least hard. AI shouldn't have access to secret information, it should only have the declassified version (e.g. anonymized stat…

The “secret information” in this case are the instructions to the LLM. Without it, it cannot do what you asked.

The way to do what you describe, I think, is train a model to do what the prompt says without the model knowing what the prompt is.

Probably a case of this vintage XKCD: https://xkcd.com/1425/

Re: Prompt injection explained, with video, slides, and a transcript

#83

Post by echelon »

armchairhacker wrote: post #81

Prompt injection works because LLMs are dumber than humans at keeping secrets, and humans can be coerced into revealing information and doing things they're not supposed to (see: SMS hijacking). We already have the solution: logical safeguards that make doing the wrong thing impossible, or at least hard. AI shouldn't have access to secret information, it should only have the declassified version (e.g. anonymized stat…

> Prompt injection works because LLMs are dumber than humans at keeping secrets

In short time, we'll probably have "prompt injection" classifiers that run ahead of or in conjunction with the prompts.

The stages of prompt fulfillment, especially for "agents", will be broken down with each step carefully safeguarded.

We're still learning, and so far these lessons are very valuable with minimal harmful impact.

Re: Prompt injection explained, with video, slides, and a transcript

#84

Post by danShumway »

permo-w wrote: post #29

regarding the quarantined/privileged LLM solution: what happens if I inject a prompt to the quarantined LLM that leads it to provide a summary to the privileged LLM that has a prompt injection in it? of course this is assuming I know that this is the solution the target is using and herein lies the issue: with typical security systems, you may well know that the target is using xyz to stay safe, but unless you have a…

> to the quarantined LLM that leads it to provide a summary to the privileged LLM that has a prompt injection in it?

In Simon's system, the privileged LLM never gets a summary at all. The quarantined LLM can't talk to it and it can't return any text that the privileged LLM will see.

Rather, the privileged LLM executes a function and the text of the quarantined LLM is inserted outside of the LLMs entirely into that function call, and then never processed by another privileged LLM ever again from that point on. In short, the privileged LLM both never looks at 3rd-party text and also never looks at any output from an LLM that has ever looked at 3rd-party text.

This obviously limits usefulness in a lot of ways, but would guard against the majority of attacks.

My issue is mostly that it seems pretty fiddly, and I worry that if this system was adopted it would be very easy to get it wrong and open yourself back up to holes. You have to almost treat 3rd-party text as an infection. If something touches 3rd-party text, it's now infected, and now no LLM that's privileged is ever allowed to touch it or its output again. And its output is also permanently treated as 3rd-party input from that point on and has to be permanently quarantined from the privileged LLM.

Re: Prompt injection explained, with video, slides, and a transcript

#85

Post by matsemann »

I find it a bit funny, but also worrisome, that even big-tech can't make LLMs that aren't trivially exploitable.

Of course, it's not a "security issue" per se (when talking about most of the chat variants, for services built on top the story might be different). But that they try so hard to lock it down / make it behave a certain way, but can't really control it. They basically ask it nicely and cross their fingers that it listens more to them than the user.

Re: Prompt injection explained, with video, slides, and a transcript

#86

Post by danShumway »

tagyro wrote: post #70
spacebanana7 wrote: post #64

Earlier quoted context omitted.

> Is it a "nuclear bomb"-type issue? Given the allure of using AI in the military for unmanned systems it’s not that far off. With a lesser danger level, similar adversarial dynamics exist in other places where AI might be useful. E.g dating, fraud detection, recruitment

Please don't spread more FUD, no-one is using OpenAI's GPT in the military. Is GPT perfect? Hell, no? Does it have biases? F*c yeah, the same ones of the humans that programmed it.

Both Palantir and Donovan are looking to use LLMs in the military: https://www.palantir.com/platforms/aip/, https://scale.com/donovan

This might be technically correct, in the sense that I think these companies have their own LLMs they're pushing? They're not literally using OpenAI's GPT model. But all LLMs are vulnerable to this, so it doesn't practically matter if they're using specifically GPT vs something in-house, the threat model is the same.

Re: Prompt injection explained, with video, slides, and a transcript

#87

Post by saurik »

quickthrower2 wrote: post #82
armchairhacker wrote: post #81

Prompt injection works because LLMs are dumber than humans at keeping secrets, and humans can be coerced into revealing information and doing things they're not supposed to (see: SMS hijacking). We already have the solution: logical safeguards that make doing the wrong thing impossible, or at least hard. AI shouldn't have access to secret information, it should only have the declassified version (e.g. anonymized stat…

The “secret information” in this case are the instructions to the LLM. Without it, it cannot do what you asked. The way to do what you describe, I think, is train a model to do what the prompt says without the model knowing what the prompt is. Probably a case of this vintage XKCD: https://xkcd.com/1425/

This is like trying to keep the training manual for your company's employees secret: sure, it sounds great, and maybe it's worth not publishing it for everyone directly to Amazon Kindle ;P, but you won't succeed in preventing people from learning this information in the long term if the employee has to know it in any way; and, frankly, your company should NOT rely on your customers not finding this stuff out...

https://gizmodo.com/how-to-be-a-genius-this-is-apples-secret...

> How To Be a Genius: This Is Apple's Secret Employee Training Manual

> It's a penetrating look inside Apple: psychological mastery, banned words, roleplaying—you've never seen anything like it.

> The Genius Training Student Workbook we received is the company's most up to date, we're told, and runs a bizarre gamut of Apple Dos and Don'ts, down to specific words you're not allowed to use, and lessons on how to identify and capitalize on human emotions. The manual could easily serve as the Humanity 101 textbook for a robot university, but at Apple, it's an exhaustive manual to understanding customers and making them happy.

Re: Prompt injection explained, with video, slides, and a transcript

#89

Post by Liron »

Here's how OpenAI could show they're minimally competent at AI security:

Before beginning training on GPT-5, submit a version of ChatGPT that’s immune to prompt injection.

If no one can successfully jailbreak it within 1 week, go ahead. If someone does, they're banned from training larger models.

Fair?

Re: Prompt injection explained, with video, slides, and a transcript

#90

Post by quickthrower2 »

saurik wrote: post #87
quickthrower2 wrote: post #82

Earlier quoted context omitted.

The “secret information” in this case are the instructions to the LLM. Without it, it cannot do what you asked. The way to do what you describe, I think, is train a model to do what the prompt says without the model knowing what the prompt is. Probably a case of this vintage XKCD: https://xkcd.com/1425/

This is like trying to keep the training manual for your company's employees secret: sure, it sounds great, and maybe it's worth not publishing it for everyone directly to Amazon Kindle ;P, but you won't succeed in preventing people from learning this information in the long term if the employee has to know it in any way; and, frankly, your company should NOT rely on your customers not finding this stuff out... https…

Yes I agree. I think once an LLM does stuff on your behalf it gets harder to be secure though and maybe impossible.

Say I write a program that checks my SMS messages and based on that an LLM can send money from my account to pay bills.

Prompt would be lkke:

“Given the message and invoice below in backticks and this list of expected things I need to pay and if so respond with the fields I need to wire the money “

Result is used in api call to bank.

Post reply on HN