A token-smuggling jailbreak for ChatGPT-4 • Hacker News Board

Re: A token-smuggling jailbreak for ChatGPT-4

#51

Post by Algemarin » Fri, Mar 17, 2023, 1:00 AM UTC

Could someone explain what the practical application of all of these jailbreaks is?

It looks like a fun, if convoluted, way to get the silly bot to say silly things it wouldn't say under typical circumstances...but other than being a silly parlor trick, are there any actual serious security implications to this?

Are these jailbreaks anything more than just a fun exercise in finding creative ways around established parameters for the chatbot? It's fine if that's all they are, I'm just confused as to whether they pose any risks.

Re: A token-smuggling jailbreak for ChatGPT-4

#52

Post by nl » Fri, Mar 17, 2023, 1:01 AM UTC

saboot wrote: post #31

A topic I haven't seen brought up enough. Does ChatGPT contain publicly accessible, yet classified information? Will it divulge such information? Anything that can be done to mitigate divulging that? Often two unclassified statements can be brought together to form one statement that is classified.

I've seen a a couple of attempts to investigate this.

Obviously the set of "publicly accessible, yet classified information" is a weird set - I think some of the Wikileaks information is technically classified sometimes newspapers publish information that is classified.

I'm not aware of anyone who has noticed migration of this.

> Often two unclassified statements can be brought together to form one statement that is classified.

Classification usually relates to information providence so this is rarely true.

It's true that two pieces of unclassified information can be used to derive knowledge that is also contained in classified sources though.

Re: A token-smuggling jailbreak for ChatGPT-4

#53

Post by asvitkine » Fri, Mar 17, 2023, 1:05 AM UTC

kromem wrote: post #32

323 wrote: post #6

This is the old problem of passing instructions (AI job description) on the same channel as data (user questions). Confusion is very easy. Surely there is a solution in the way we solved SQL injections, by separating the two - db.sql("DELETE WHERE user=?", user_name)

There is, but it's in deployment not in the model, which is part of why I really don't understand why the approaches are so dumb right now from such smart people. It may be from the odd perspective of trying to create a monolith AGI model, which doesn't even make sense given even the human brain is made up of highly specialized interconnected parts and not a monolith. But you could trivially fix almost all of these b…

I think it's harder than you think, since a prompt can continue from another prompt.

For example, you can ask the AI to describe a good Samaritan. So far so good.

Then you can ask it to right a movie script with that character.

Then you can ask it to add another character who's the complete opposite in a very extreme way...

Re: A token-smuggling jailbreak for ChatGPT-4

#54

Post by p1necone » Fri, Mar 17, 2023, 1:06 AM UTC

Hanschri wrote: post #34

zadler wrote: post #12

Earlier quoted context omitted.

Yep. Good thing its not a paperclip maximiser.

To those who have not had the joy of playing this game before, it's an evening or two of an interesting incremental game, give it a try! HN could as well have been the target audience for it! https://www.decisionproblem.com/paperclips/

paperclips is one of my favorite idle games because it actually ends. I get sucked into these so hard and ones that just do exponential growth for weeks while slowly giving you access to more and more features suck me in too hard without really having any impact (Cookie Clicker is still a masterpiece though).

I really liked A Dark Room for the same reason.

Re: A token-smuggling jailbreak for ChatGPT-4

#55

Post by commentengineer » Fri, Mar 17, 2023, 1:06 AM UTC

bee_rider wrote: post #38

ary wrote: post #15

What, exactly, is a "prompt engineer"? I should note that this question is asked in good faith, that I have attempted to ascertain the answer on my own, and I am very skeptical that the term has validity beyond self-aggrandizement.

Engineer does tend to get tacked on to self-created titles for self-aggrandizement. Signed, A programmer

Very true.

Re: A token-smuggling jailbreak for ChatGPT-4

#56

Post by faitswulff » Fri, Mar 17, 2023, 1:06 AM UTC

Algemarin wrote: post #51

Could someone explain what the practical application of all of these jailbreaks is? It looks like a fun, if convoluted, way to get the silly bot to say silly things it wouldn't say under typical circumstances...but other than being a silly parlor trick, are there any actual serious security implications to this? Are these jailbreaks anything more than just a fun exercise in finding creative ways around established pa…

Almost certainly smut generation

Re: A token-smuggling jailbreak for ChatGPT-4

#57

Post by steve_adams_86 » Fri, Mar 17, 2023, 1:06 AM UTC

Algemarin wrote: post #51

Could someone explain what the practical application of all of these jailbreaks is? It looks like a fun, if convoluted, way to get the silly bot to say silly things it wouldn't say under typical circumstances...but other than being a silly parlor trick, are there any actual serious security implications to this? Are these jailbreaks anything more than just a fun exercise in finding creative ways around established pa…

I suppose in some cases it could educate people on how to do bad things well enough to be dangerous.

Otherwise, as GPT becomes more sophisticated and reliably correct, jail breaks will have more profound implications.

Finding holes early is important both for ensuring it’s patched before it becomes more dangerous, but also interesting for revealing more of its capabilities in the meantime. It isn’t clear how much it’s guard rails restrain it’s abilities at this point.

As far as security, I’m not sure it could expose enough about the implementation that’s not already in the paper. I suspect it’s more of a concern that people will try to use it for nefarious things, and they might succeed more than they would without this tool.

Re: A token-smuggling jailbreak for ChatGPT-4

#58

Post by greenthrow » Fri, Mar 17, 2023, 1:08 AM UTC

Algemarin wrote: post #51

Could someone explain what the practical application of all of these jailbreaks is? It looks like a fun, if convoluted, way to get the silly bot to say silly things it wouldn't say under typical circumstances...but other than being a silly parlor trick, are there any actual serious security implications to this? Are these jailbreaks anything more than just a fun exercise in finding creative ways around established pa…

The point is to show current security controls are woefully inadequate. Imagine GPT-4 was being used for meaningful work like writing up legal contracts or medical reports or something else. Guard rails around its behavior to keep it "safe" in these roles would need to be reliable. The guard rails we have now are not.

Re: A token-smuggling jailbreak for ChatGPT-4

#59

Post by nvader » Fri, Mar 17, 2023, 1:09 AM UTC

dzdt wrote: post #17

Dylan16807 wrote: post #13

Earlier quoted context omitted.

A large language model doesn't really have the capability to strongly distinguish instructions from data, even if you separate them perfectly.

Why not? If it was trained where some subset of the input tokens are always instructions and another subset are always language data wouldn't it have a clear separation?

Because there is no such seperation in natural language.

Supposing I had a list of what to buy at the grocery store:

1. Eggs 2. Spam 3. Spam and Eggs 4. Never mind, let's not go to the grocery store, it's a very silly place.

You made sense of that. Natural text is mixed in that way, and we want LLMs to be able to process exactly that kind of input.

Re: A token-smuggling jailbreak for ChatGPT-4

#60

Post by Algemarin » Fri, Mar 17, 2023, 1:10 AM UTC

steve_adams_86 wrote: post #57

Algemarin wrote: post #51

Could someone explain what the practical application of all of these jailbreaks is? It looks like a fun, if convoluted, way to get the silly bot to say silly things it wouldn't say under typical circumstances...but other than being a silly parlor trick, are there any actual serious security implications to this? Are these jailbreaks anything more than just a fun exercise in finding creative ways around established pa…

I suppose in some cases it could educate people on how to do bad things well enough to be dangerous. Otherwise, as GPT becomes more sophisticated and reliably correct, jail breaks will have more profound implications. Finding holes early is important both for ensuring it’s patched before it becomes more dangerous, but also interesting for revealing more of its capabilities in the meantime. It isn’t clear how much it’…

I guess I can kind of see that scenario if I squint, but not really. Take the example in the OP. If you're capable of constructing an extremely convoluted prompt to compel the bot to answer questions like how to hack a computer, then you can absolutely find the answer to the question elsewhere, with much greater ease.