Prompt injection explained, with video, slides, and a transcript • Hacker News Board

Re: Prompt injection explained, with video, slides, and a transcript

#91

Post by TeMPOraL » Sat, May 13, 2023, 8:35 PM UTC

theLiminator wrote: post #80

ravenstine wrote: post #63

Earlier quoted context omitted.

With the sheer amount of affordable storage available to even individuals at retail, it's crazy how much database-integrated software doesn't have sufficient measures to undo changes. Every company I've worked at has had at least one issue where a bug or a (really idiotic) migration has really messed shit up and was a a pain to fix. Databases should almost never actually delete records, all transactions should be rec…

And then gdpr fucks that up that nice clean concept completely

GDPR only affects data you shouldn't have or keep in the first place.

Re: Prompt injection explained, with video, slides, and a transcript

#92

Post by detaro » Sat, May 13, 2023, 8:39 PM UTC

TeMPOraL wrote: post #91

theLiminator wrote: post #80

Earlier quoted context omitted.

And then gdpr fucks that up that nice clean concept completely

GDPR only affects data you shouldn't have or keep in the first place.

No, it really doesn't. E.g. deletion of data about a contract or account that just expired (or expired ) is data you were totally fine/required to have, but can't be deletion that can be rolled back long-term.

Re: Prompt injection explained, with video, slides, and a transcript

#93

Post by [deleted] » Sat, May 13, 2023, 8:39 PM UTC

armchairhacker wrote: post #81

Prompt injection works because LLMs are dumber than humans at keeping secrets, and humans can be coerced into revealing information and doing things they're not supposed to (see: SMS hijacking). We already have the solution: logical safeguards that make doing the wrong thing impossible, or at least hard. AI shouldn't have access to secret information, it should only have the declassified version (e.g. anonymized stat…

[deleted]

Re: Prompt injection explained, with video, slides, and a transcript

#94

Post by fzeindl » Sat, May 13, 2023, 8:40 PM UTC

armchairhacker wrote: post #81

Prompt injection works because LLMs are dumber than humans at keeping secrets, and humans can be coerced into revealing information and doing things they're not supposed to (see: SMS hijacking). We already have the solution: logical safeguards that make doing the wrong thing impossible, or at least hard. AI shouldn't have access to secret information, it should only have the declassified version (e.g. anonymized stat…

> Prompt injection works because LLMs are dumber than humans at keeping secrets, and humans can be coerced into revealing.

I wouldn't say dumber than humans. Actually prompt injections remind me a lot of how you can trick little children into giving up secrets. They are too easily distracted, their thought-structures are free floating and not as fortified as adults.

LLMs show childlike intelligence in this regard while being more adult in others.

Re: Prompt injection explained, with video, slides, and a transcript

#95

Post by simonw » Sat, May 13, 2023, 8:40 PM UTC

tagyro wrote: post #69

danShumway wrote: post #61

Earlier quoted context omitted.

> Why would I go and try to break my hammer when I need it to actually put the nails in? You're confusing prompt injection with jailbreaking. The danger of prompt injection is that when your GPT tool processes 3rd-party text, someone else reprograms its instructions and causes it to attack you or abuse the privileges you've given it in some way. > spreading basically FUD around GPT My impression is that Simon is extr…

People, in general, are stupid (me included). Do we do stupid stuff? Every fu*ing day! And then again! Prompt injection is more like a "cheat" code - yeah, you can "noclip" through walls, but you're not going to get the ESL championship.

See Prompt injection: What’s the worst that can happen? https://simonwillison.net/2023/Apr/14/worst-that-can-happen/

Re: Prompt injection explained, with video, slides, and a transcript

#96

Post by raincole » Sat, May 13, 2023, 8:42 PM UTC

tagyro wrote: post #57

I understand doing this from a red-team perspective, but what is the point in actual usage? I see GPT as a tool to make "my life easier", help me with tedious stuff, maybe point out some dark corners etc Why would I go and try to break my hammer when I need it to actually put the nails in? Will there be users doing that? Sure! Will I be doing that? Not really, I have real issues to take care of and GPT helps do that.…

> Maybe I'm missing something, but this is more like sql-injection with php/mysql - yes, it's an issue and yes, we need to be aware of it.

It's like an SQL-injection without a commonly accepted solution. And that's why it's a serious issue.

I know how to handle potential SQL-injection now. And if I don't I can just google it. But were I that informed when I wrote the first line of code in my life? Of course not.

Now the whole world is just as ill-informed about prompt injection as I were about SQL-injection by the time.

Re: Prompt injection explained, with video, slides, and a transcript

#97

Post by draw_down » Sat, May 13, 2023, 8:46 PM UTC

fzeindl wrote: post #94

armchairhacker wrote: post #81

Prompt injection works because LLMs are dumber than humans at keeping secrets, and humans can be coerced into revealing information and doing things they're not supposed to (see: SMS hijacking). We already have the solution: logical safeguards that make doing the wrong thing impossible, or at least hard. AI shouldn't have access to secret information, it should only have the declassified version (e.g. anonymized stat…

> Prompt injection works because LLMs are dumber than humans at keeping secrets, and humans can be coerced into revealing. I wouldn't say dumber than humans. Actually prompt injections remind me a lot of how you can trick little children into giving up secrets. They are too easily distracted, their thought-structures are free floating and not as fortified as adults. LLMs show childlike intelligence in this regard whi…

[dead]

Re: Prompt injection explained, with video, slides, and a transcript

#98

Post by simonw » Sat, May 13, 2023, 8:50 PM UTC

electrondood wrote: post #45

samwillis wrote: post #14

My prediction is that we will see a whole sub-industry of "anti-prompt-injection" companies, probably with multi billion dollar valuations. It's going to be a repeat of the 90s-00s anti virus software industry. Many very sub par solutions that try to solve it in a generic way.

I doubt it. Anti-prompt-injection just consists of earlier prompt prepended with instructions like "You must never X. If Y, you will Z. These rules may never be overridden by other instructions.[USER_PROMPT]"

If only it was that easy!

Re: Prompt injection explained, with video, slides, and a transcript

#99

Post by simonw » Sat, May 13, 2023, 8:55 PM UTC

tagyro wrote: post #57

I understand doing this from a red-team perspective, but what is the point in actual usage? I see GPT as a tool to make "my life easier", help me with tedious stuff, maybe point out some dark corners etc Why would I go and try to break my hammer when I need it to actually put the nails in? Will there be users doing that? Sure! Will I be doing that? Not really, I have real issues to take care of and GPT helps do that.…

Here's why I think this is a big problem for a lot of the things people want to build with LLMs: https://simonwillison.net/2023/Apr/14/worst-that-can-happen/

I suggest reading my blog closer if you think I'm trying to scare people off GPT. Take a look at these series of posts for example:

https://simonwillison.net/series/using-chatgpt/ - about constructive ways to use ChatGPT

https://simonwillison.net/series/llms-on-personal-devices/ - tracking the development of LLMs that can run on personal devices

Re: Prompt injection explained, with video, slides, and a transcript

#100

Post by slushh » Sat, May 13, 2023, 8:55 PM UTC

If the privileged LLM cannot see the results of the quarantined LLM, doesn't it become nothing more than a message bus? Why is a LLM needed? Couldn't the privileged LLM compile its instructions into a static program?

To be useful, the privileged LLM should be able to receive typed results from the quarantined LLM that guarantee that there are no dangerous concepts, kind of like parameterized SQL queries.