Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

GitHub Copilot Chat Leaked Prompt

twitter.com

Post reply on HN
Print view View original

511–520 of 628 posts

Re: GitHub Copilot Chat Leaked Prompt

#511

Post by jakelazaroff »

jameshart wrote: post #437
tinco wrote: post #233

Earlier quoted context omitted.

You had the right understanding in your first comment, but what was missing was the fine tuning. You are right that there aren't many documents on the web that are structured that way, so the raw model wouldn't be very effective on predicting the next token. But since we know that it will complete a command when structured it cleverly, all we had to do to fine tune it is synthesize (generate) a bazillion examples of…

You’ve expressed this very well - Thank you. I get that the fine tuning is done over documents which are generated to encourage the dialog format. What I’m intrigued by is the way prompters choose to frame those documents. Because that is a choice . It’s a manufactured training set. Using the ‘you are an ai chatbot’ style of prompting, in all the samples we generate and give to the model, text attributed to {:system}…

Really thought provoking thread, and I’m glad you kept prodding at the issue. I hadn’t considered the anthropomorphism from this angle, but it makes sense — we’ve built it to respond in this way because we “want” to interact with it in this way. It really does seem like we’re striving for a very specific vision from science fiction.

That said: you can say the same thing about everything in technology. An untuned LLM might not be receptive to prompting in this way, but an LLM is also an entirely human invention — i.e. a choice. There’s not really any aspect of technology that isn’t based on our latent desires/fears/etc. The LLM interface definitely has the biggest uncanny valley though.

Re: GitHub Copilot Chat Leaked Prompt

#512

Post by wongarsu »

pixl97 wrote: post #504
wongarsu wrote: post #482

Earlier quoted context omitted.

There probably are solutions to this problem, we just haven't found them yet. Bing chat uses [system] [user] and [assistant] to differentiate the sections, and that seems to have some effect (most notably when they forgot to filter [system] in webpages, allowing websites that the chatbot was looking at to reprogram the chatbot). Some people suggested just making those special tokens that can't be produced from normal…

>just making those special tokens that can't be produced from normal text Heh, from the world of HTTP filtering in 'dumb' contexts we still run into situations in mature software where we find escapes that lead to exploits. In LLMs is possible it could be far harder to prevent these special tokens from being accessed. Just as a play idea. Lets say the system prompt is defined by the character with identity '42' that…

Even taking a simple system that is supposed to summarize long texts that might exceed the context size: the simple approach is to cut the document into segments, have the LLM summarize each segment separately, then generate a summary of those summaries. Now you have to defend against attacks not just from the original text, but also from the intermediate summaries (which are indirectly under attacker control). Which is only going to get worse as we add more opportunities for internal thought to our models, which also has to be protected.

It's like defending against SQL injection before parameterized statements were invented. Forget calling real_escape_string(input) once in your entire codebase, and the attacker owns your system.

Re: GitHub Copilot Chat Leaked Prompt

#513

Post by pixl97 »

radres wrote: post #296
htag wrote: post #117

Earlier quoted context omitted.

Sure, the prompt is bland. The interesting sauce is GPT4 cannot keep a secret. If you have a GPT4 powered user interface be sure not to load it with context you do not want directly leaking to the user.

I made a twitter bot and specifically told the bot to act like a human, and be bit of an asshole (to gain more interactions). Then someone got angry and asked "Who are you!!!" and my bot replied "I am an AI language model..." I mean, come on GPT.

> I mean, come on GPT.

OpenAI would consider that a success at least at this point. They don't want the bot pretending to be a human at this point.

Re: GitHub Copilot Chat Leaked Prompt

#514

Post by jameshart »

mxkopy wrote: post #206
jameshart wrote: post #149

Earlier quoted context omitted.

Right. But who's the 'you' who's being addressed by the {:system} prompt? Who is the {:assistant} supposed to think the {:system} is? Why should the {:assistant} output tokens that make it do what the {:system} tells it to? After all, the {:user} doesn't. The {:system} doesn't provide any instructions for how the {:user} is supposed to behave, the {:user} tokens are chosen arbitrarily and don't match the probabilitie…

If I'm feeling romantic I think about a universal 'you' separate from the person that is referred to and is addressed by every usage of the word - a sort of ghost in the shell that exists in language. But really, it's probably just priming the responses to fit the grammatical structure of a first person conversation. That structure probably does a lot of heavy lifting in terms of how information is organized, too, so…

Sure. We’re talking about language models so the only tools we have to work with are language after all.

Which is what gets me thinking - do we get different chatbot results from prompts that look like each of these:

  You are an AI chatbot
  Sydney is an AI chatbot
  I am an AI chatbot
  There is an AI chatbot
  Say there was an AI chatbot
  Say you were an AI chatbot
  Be an AI chatbot
  Imagine an AI chatbot
  AI chatbots exist
  This is an AI chatbot
  We are in an AI chatbot
If we do… that’s fascinating.

If we don’t… why do prompt engineers favor one form over any other here? (Although this stops being a software engineering question and becomes an anthropology question instead)

Re: GitHub Copilot Chat Leaked Prompt

#515

Post by jeffreyrogers »

rf15 wrote: post #161
jeffreyrogers wrote: post #160

Why would asking an LLM to leak its prompt actually cause it to leak its real prompt? You could ask it to leak its weights too and it would probably give you something back but no one would take that seriously. I just don't see architecturally why you should trust that a "leaked prompt" is anything but some statistically generated text, just like everything else an LLM produces.

because this prompt is part of what is applied to the model when you prompt it, its weights are not. Training is a different process from prompting, and weights are an internal property not part of the actual input in any way. E.g. it's like asking "How many brain cells do you have? I mean it's your own brain, surely you must know the exact number?"

If you put a known prompt in an LLM and ask it to read it back to you, how often does it do it correctly? I would bet not all the time, particularly if you give it a long prompt like the one that is proposed here.

Re: GitHub Copilot Chat Leaked Prompt

#516

Post by logicallee »

logicallee wrote: post #290
otikik wrote: post #227

Earlier quoted context omitted.

You are anthropomorphing. The machine doesn’t “really” understand, it’s just “simulating” it understands. “You” is “3 characters on an input string that are used to configure a program”. The prompt could have been any other thing, including a binary blob. It’s just more convenient for humans to use natural language to communicate, and the machine already has natural language features, so they used that instead of cre…

Here is how you can know that ChatGPT really understands, rather than simulating that it understands: - You can give it specific instructions and it will follow them, modifying its behavior by doing so. This shows that the instructions are understood well enough to be followed. For example, if you ask it to modify its behavior by working through its steps, then it will modify its behavior to follow your request. This…

People are downvoting me, so I'll add a counterexample: suppose you teach your dog to fetch your slipper to where if you say "fetch my slipper" it knows it should bring you your slipper and it does so. Does it really understand the instructions: no. So what is the difference between this behavior and true understanding? How can one know it doesn't truly understand?

Well, if you change your instructions to be more complicated it fails immediately. If you say "I have my left shoe bring me the other one" it could not figure out that "the other one" is the right shoe, even if it were labelled. Basically it can't follow more complicated instructions, which is how you know it doesn't really understand them.

Unlike the dog, GPT 4 modifies its behavior to follow more complicated instructions as well. Not as well as humans, but well enough to pass a bar exam that isn't in its training set.

Re: GitHub Copilot Chat Leaked Prompt

#517

Post by jameshart »

jameshart wrote: post #503
Name_Chawps wrote: post #494

Earlier quoted context omitted.

I have done some prompt engineering and read about prompt engineering, and I believe people write in the imperative mood because they have tried different ways of doing it and they believe it gives better results. I.e., this practice is informed by trial and error, not theory.

They’re not writing in the imperative mood. An imperative prompt would read: Be an ai chatbot Be kind and helpful and patient … But at that point the text prediction would probably devolve into 4chan green text nonsense so it’s probably best not to go there.

And to complete the thought:

The ‘You are an AI chatbot’ form is actually grammatically ‘predicative’, not ‘imperative’ (ie it describes what is not what must be done)

Re: GitHub Copilot Chat Leaked Prompt

#519

Post by ImHereToVote »

nayuki wrote: post #499
pixl97 wrote: post #484

Earlier quoted context omitted.

Get back to me when the MP3 has a few billion words (songs?) it can choose from, and when you walk into the room with it and say 'howdy' it responds correctly with 'hello' back.

The good ol' https://en.wikipedia.org/wiki/Chinese_room argument ... with audio files!

Except the chinese room creates a model that can create uniqe answers.

Re: GitHub Copilot Chat Leaked Prompt

#520

Post by emodendroket »

pmoriarty wrote: post #378
emodendroket wrote: post #145

Earlier quoted context omitted.

Right, but the concern is that bias can show up in more insidious ways.

as it can in humans

Nobody disputes that but AI can give a false impression of being less biased than it's creators when in fact it has just codified their biases.
Post reply on HN