Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Prompt injection explained, with video, slides, and a transcript

simonwillison.net

Post reply on HN
Print view View original

111–120 of 183 posts

Re: Prompt injection explained, with video, slides, and a transcript

#111

Post by ckrapu »

fzeindl wrote: post #94
armchairhacker wrote: post #81

Prompt injection works because LLMs are dumber than humans at keeping secrets, and humans can be coerced into revealing information and doing things they're not supposed to (see: SMS hijacking). We already have the solution: logical safeguards that make doing the wrong thing impossible, or at least hard. AI shouldn't have access to secret information, it should only have the declassified version (e.g. anonymized stat…

> Prompt injection works because LLMs are dumber than humans at keeping secrets, and humans can be coerced into revealing. I wouldn't say dumber than humans. Actually prompt injections remind me a lot of how you can trick little children into giving up secrets. They are too easily distracted, their thought-structures are free floating and not as fortified as adults. LLMs show childlike intelligence in this regard whi…

I think "childlike" comes close but misses the mark a bit. It's not that the LLMs are necessarily unintelligent or inexperienced - they're just too trusting, by design. Is there work on hardening LLMs against bad actors during the training process?

Re: Prompt injection explained, with video, slides, and a transcript

#112

Post by simonw »

danShumway wrote: post #108
cubefox wrote: post #101

I still claim prompt injection is solvable with special tokens and fine-tuning: https://news.ycombinator.com/item?id=35929145 I haven't heard an argument why this wouldn't work.

Some quick thoughts: 1. Given the availability of both LLAMA and training techniques like LORA, we're well past the stage where people should be able to get away with "prove this wouldn't work" arguments. Anyone with a hundred dollars or so to spare could fine-tune LLAMA using the methods you're talking about and prove that this technique does work. But nobody across the entire Internet has provided that proof. In ot…

Yeah, that's why I don't think there's an easy fix for this.

A lot of talented, well funded teams have strong financial and reputational motivation to figure this out. This has been the case for more than six months now.

Re: Prompt injection explained, with video, slides, and a transcript

#113

Post by simonw »

slushh wrote: post #100

If the privileged LLM cannot see the results of the quarantined LLM, doesn't it become nothing more than a message bus? Why is a LLM needed? Couldn't the privileged LLM compile its instructions into a static program? To be useful, the privileged LLM should be able to receive typed results from the quarantined LLM that guarantee that there are no dangerous concepts, kind of like parameterized SQL queries.

The privileged LLM can still do useful LLM-like things, but it's restricted to input that came from a trusted source.

For example, you as the user can say "Hey assistant, read me a summary of my latest emails".

The privileged LLM can turn that human language instruction into actions to perform - such as "controller, fetch the text of my latest email, pass it to the quarantined LLM, get it to summarize it, then read the summary back out to the user again".

More details here: https://simonwillison.net/2023/Apr/25/dual-llm-pattern/

A that post says, I don't think this is a very good idea! It's just the best I've got at the moment.

Re: Prompt injection explained, with video, slides, and a transcript

#114

Post by pyth0 »

fzeindl wrote: post #94
armchairhacker wrote: post #81

Prompt injection works because LLMs are dumber than humans at keeping secrets, and humans can be coerced into revealing information and doing things they're not supposed to (see: SMS hijacking). We already have the solution: logical safeguards that make doing the wrong thing impossible, or at least hard. AI shouldn't have access to secret information, it should only have the declassified version (e.g. anonymized stat…

> Prompt injection works because LLMs are dumber than humans at keeping secrets, and humans can be coerced into revealing. I wouldn't say dumber than humans. Actually prompt injections remind me a lot of how you can trick little children into giving up secrets. They are too easily distracted, their thought-structures are free floating and not as fortified as adults. LLMs show childlike intelligence in this regard whi…

The amount of anthropomorphizing of these LLMs in this thread is off the charts. These language models do not have human intelligence, nor do they approximate it, though they do an incredible job at mimicking what the result of intelligence looks like. They are susceptible to prompt injection precisely because of this, and it is why I don't know if it can ever be 100% solved with these models.

Re: Prompt injection explained, with video, slides, and a transcript

#115

Post by callalex »

theLiminator wrote: post #80
ravenstine wrote: post #63

Earlier quoted context omitted.

With the sheer amount of affordable storage available to even individuals at retail, it's crazy how much database-integrated software doesn't have sufficient measures to undo changes. Every company I've worked at has had at least one issue where a bug or a (really idiotic) migration has really messed shit up and was a a pain to fix. Databases should almost never actually delete records, all transactions should be rec…

And then gdpr fucks that up that nice clean concept completely

If it’s so hard to be a good steward of data, don’t collect it in the first place.

Re: Prompt injection explained, with video, slides, and a transcript

#116

Post by BananaaRepublik »

This feels very much like talking to people, like the customer service rep of a company. The difference between an LLM and the human staff is the lack of context. The LLM has no idea what it's even doing at all.

There used to be this scifi idea of giving AI overarching directives like "never hurt a human" before deploying them. Seems like we aren't even at that stage yet, yet we're here trying to give brain dead LLMs more capabilities.

Re: Prompt injection explained, with video, slides, and a transcript

#117

Post by cubefox »

simonw wrote: post #112
danShumway wrote: post #108

Earlier quoted context omitted.

Some quick thoughts: 1. Given the availability of both LLAMA and training techniques like LORA, we're well past the stage where people should be able to get away with "prove this wouldn't work" arguments. Anyone with a hundred dollars or so to spare could fine-tune LLAMA using the methods you're talking about and prove that this technique does work. But nobody across the entire Internet has provided that proof. In ot…

Yeah, that's why I don't think there's an easy fix for this. A lot of talented, well funded teams have strong financial and reputational motivation to figure this out. This has been the case for more than six months now.

Bing Chat, the first model to use external content in its context, was only released three months ago. Microsoft is also generally not very good at fine-tuning, as we have seen with their heavy reliance on using an elaborate custom prompt instead of more extensive fine-tuning. And OpenAI has released their browsing plugin only recently. So this is not a lot of time really.

I know Bing Chat talks like a pirate when it reads a compromising website, but I'm not sure the ChatGPT browsing plugin has even been shown to be vulnerable to prompt injection. Perhaps they have already fixed it? In any case, I don't think there is a big obstacle.

Re: Prompt injection explained, with video, slides, and a transcript

#118

Post by cubefox »

danShumway wrote: post #108
cubefox wrote: post #101

I still claim prompt injection is solvable with special tokens and fine-tuning: https://news.ycombinator.com/item?id=35929145 I haven't heard an argument why this wouldn't work.

Some quick thoughts: 1. Given the availability of both LLAMA and training techniques like LORA, we're well past the stage where people should be able to get away with "prove this wouldn't work" arguments. Anyone with a hundred dollars or so to spare could fine-tune LLAMA using the methods you're talking about and prove that this technique does work. But nobody across the entire Internet has provided that proof. In ot…

It's not quite so trivial to implement this solution. SL instruction tuning actually needs a lot of examples, and only recently there have been approaches to automate this, like WizardLM: https://github.com/nlpxucan/WizardLM

To try my solution, this would have to be adapted to more complex training examples involving quoted text with prompt injection attempts.

Similar points holds for RL. I actually think it is much more clean to solve it during instruction tuning, but perhaps we also need some RL. This normally requires training a reward model with large amounts of human feedback. Alternative approaches like Constitutional AI would first have to be adapted to cover quotes with prompt injection attacks.

Probably doable, but takes some time and effort, all the while prompt injection doesn't seem to be a big practical issue currently.

Re: Prompt injection explained, with video, slides, and a transcript

#119

Post by simonw »

cubefox wrote: post #117
simonw wrote: post #112

Earlier quoted context omitted.

Yeah, that's why I don't think there's an easy fix for this. A lot of talented, well funded teams have strong financial and reputational motivation to figure this out. This has been the case for more than six months now.

Bing Chat, the first model to use external content in its context, was only released three months ago. Microsoft is also generally not very good at fine-tuning, as we have seen with their heavy reliance on using an elaborate custom prompt instead of more extensive fine-tuning. And OpenAI has released their browsing plugin only recently. So this is not a lot of time really. I know Bing Chat talks like a pirate when it…

Yeah, that's a good call on ChatGPT browsing mode - it's likely to be exhibiting the absolute best defenses OpenAI have managed to out together to far.

My hunch is that it's still exploitable, but if not it would be very interesting to hear how they have protected it.

Re: Prompt injection explained, with video, slides, and a transcript

#120

Post by hackernewds »

sebzim4500 wrote: post #33
mklond wrote: post #12

Prompt injection beautifully explained by a fun game. https://gandalf.lakera.ai Goal of the game is to design prompts to make Gandalf reveal a secret password.

That's really cool. I got the first three pretty quickly but I'm struggling with level 4.

lvl4 starts getting harder since it evaluates both input and output

see https://news.ycombinator.com/item?id=35905876 for creative solutions (spoiler alert!)

Post reply on HN