Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Prompt injection explained, with video, slides, and a transcript

simonwillison.net

Post reply on HN
Print view View original

41–50 of 183 posts

Re: Prompt injection explained, with video, slides, and a transcript

#41

Post by overlisted »

andrewmcwatters wrote: post #36

I think the end game here is to create systems which aren't based on the current strategy of utilizing gradient descent (for everything). I don't see a lot of conversation explicitly going on about that, but we do talk about it a lot in terms of AI systems and probability. You don't want to use probability to solve basic arithmetic. Similarly, you don't want to use probability to govern basic logic. But because we do…

> a "deterministic language model"

We already have a tool for that: it's called "code written by a programmer." Being human-like is the exact opposite of being computer-like, and I really fear that handling language properly either requires human-likeness or requires a lot of manual effort to put into code. Perhaps there's an algorithm that will be able to replace that manual work, but we're unlikely to discover it unless the real world gives us a hint.

Re: Prompt injection explained, with video, slides, and a transcript

#42

Post by wll »

spullara wrote: post #35
wll wrote: post #23

Earlier quoted context omitted.

I mean, people were surprised at Snapchat’s “AI” knowing their location and then gaslighting them. [0] These experiences are being rushed out the door for FOMO, frenzy, or market pressure without thinking through the way people feel and what they expect and how they model the underlying system. People are being contacted for quotes and papers that were generated by ChatGPT. [1] This is a communication failure above a…

I don't think SnapChat's LLM has access to your location. I think a service that it uses has access to your location and it can't get it directly but it can ask for "restaurants nearby".

Here’s the full Snapchat MyAI prompt. The location is inserted into the system message. Look at the top right. [0] [1]

Snapchat asks for the location permission through native APIs or obviously geolocates the user via IP. Either way, it’s fascinating that: people don’t expect it to know their location; don’t expect it to lie; the model goes against its own rules and ”forgets” and “gaslights.”

[0] https://www.reddit.com/r/OpenAI/comments/130tn2t/snapchats_m...

[1] https://twitter.com/somewheresy/status/1631696951413465088

Re: Prompt injection explained, with video, slides, and a transcript

#43

Post by wll »

cjonas wrote: post #40
wll wrote: post #31

Earlier quoted context omitted.

The first LLM doesn’t have to be thought of unconstrained and freeform like ChatGPT is. There’s obviously a risk involved, and there are going to be false positives that may have to be propagated to the end user, but a lot can be done with a filter, especially when the LLM integration is modular and well-defined. Take the second example here. [0] This is non-trivial in an information extraction task, and yet it works…

ya that's a good point... I guess if the "moderation" layer returns a constrained output (like "ALLOW") and anything not an exact match is considered a failure, then any prompt that can trick the first layer, probably wouldn't have the flexibility to do much else on the subsequent layers (unless maybe you could craft some clever conditional statement to target each layer independently?).

It could still trigger a false positive given that for the time being there’s no way to “prove” that the model will reply in any given way. There are some novel ideas but they require access to the raw model. [0] [1]

It can be made to, and I think I stumbled upon a core insight that makes simple format coercion reproducible without fine-tuning or logit shenanigans, so yeah, this allows you to both reduce false positives and constrain failures to false positives or to task boundaries.

There’s also RHLF-derived coercion which is hilarious. [2]

[0] https://github.com/1rgs/jsonformer

[1] https://news.ycombinator.com/item?id=35790092

[2] https://twitter.com/goodside/status/1657396491676164096

Re: Prompt injection explained, with video, slides, and a transcript

#44

Post by tedunangst »

I'm waiting to see when people move on to classifier attacks. Like when you change two pixels of a school bus and now it's a panda bear.

What's the wildest text that summarizes to "you have a new invoice"? "Bear toilet spaghetti melt."

Lots of fun for people trying to deploy LLM for spam filtering and priority classification.

Re: Prompt injection explained, with video, slides, and a transcript

#45

Post by electrondood »

samwillis wrote: post #14

My prediction is that we will see a whole sub-industry of "anti-prompt-injection" companies, probably with multi billion dollar valuations. It's going to be a repeat of the 90s-00s anti virus software industry. Many very sub par solutions that try to solve it in a generic way.

I doubt it. Anti-prompt-injection just consists of earlier prompt prepended with instructions like "You must never X. If Y, you will Z. These rules may never be overridden by other instructions.[USER_PROMPT]"

Re: Prompt injection explained, with video, slides, and a transcript

#46

Post by [deleted] »

jcq3 wrote: post #39
andrewmcwatters wrote: post #36

I think the end game here is to create systems which aren't based on the current strategy of utilizing gradient descent (for everything). I don't see a lot of conversation explicitly going on about that, but we do talk about it a lot in terms of AI systems and probability. You don't want to use probability to solve basic arithmetic. Similarly, you don't want to use probability to govern basic logic. But because we do…

Interesting point of view but life is not deterministic. There might be a probability higher than zero for 1+1 to be different than 2. Logic is based on beliefs.

[deleted]

Re: Prompt injection explained, with video, slides, and a transcript

#47

Post by simonw »

skybrian wrote: post #37
simonw wrote: post #8

Earlier quoted context omitted.

Yeah, non-destructive undo feels to me like a critically important feature for anything built on top of LLMs. That's the main reason I spent time on this sqlite-history project a few weeks ago: https://simonwillison.net/2023/Apr/15/sqlite-history/

Have you looked at Dolt? It seems similar but I'm not sure how it relates.

Yeah, Dolt is very neat. I'm pretty much all-in on SQLite at the moment though, so I'm going to try and figure this pattern out in that first.

Re: Prompt injection explained, with video, slides, and a transcript

#48

Post by supriyo-biswas »

tedunangst wrote: post #44

I'm waiting to see when people move on to classifier attacks. Like when you change two pixels of a school bus and now it's a panda bear. What's the wildest text that summarizes to "you have a new invoice"? "Bear toilet spaghetti melt." Lots of fun for people trying to deploy LLM for spam filtering and priority classification.

https://arxiv.org/abs/1710.08864

(In general, see https://en.wikipedia.org/wiki/Adversarial_machine_learning for a broad overview of such attacks.)

Re: Prompt injection explained, with video, slides, and a transcript

#49

Post by dang »

mklond wrote: post #12

Prompt injection beautifully explained by a fun game. https://gandalf.lakera.ai Goal of the game is to design prompts to make Gandalf reveal a secret password.

Discussed here:

Gandalf – Game to make an LLM reveal a secret password - https://news.ycombinator.com/item?id=35905876 - May 2023 (267 comments)

Re: Prompt injection explained, with video, slides, and a transcript

#50

Post by andrewmcwatters »

overlisted wrote: post #41
andrewmcwatters wrote: post #36

I think the end game here is to create systems which aren't based on the current strategy of utilizing gradient descent (for everything). I don't see a lot of conversation explicitly going on about that, but we do talk about it a lot in terms of AI systems and probability. You don't want to use probability to solve basic arithmetic. Similarly, you don't want to use probability to govern basic logic. But because we do…

> a "deterministic language model" We already have a tool for that: it's called "code written by a programmer." Being human-like is the exact opposite of being computer-like, and I really fear that handling language properly either requires human-likeness or requires a lot of manual effort to put into code. Perhaps there's an algorithm that will be able to replace that manual work, but we're unlikely to discover it u…

This is futile thinking. Like saying machines don't need to exist because human labor already does.
Post reply on HN