GitHub Copilot Chat Leaked Prompt • Hacker News Board

Re: GitHub Copilot Chat Leaked Prompt

#121

Post by lachlan_gray » Sat, May 13, 2023, 3:32 AM UTC

jameshart wrote: post #109

Something that I find weird about these chat prompts (assuming they are real, not hallucinated): They're almost always written in second person*. "You are an AI programming assistant" "You are about to immerse yourself into the role of another Al model known as DAN" Who are these prompts addressed to? Who does the GPT think wrote them? The thing that confuses me is that these are text token prediction algorithms, und…

I think it started off because humans are humans, and have an easier time talking to something rather than “talking something”. One purpose of RLHF is so that the models tend to work well when you speak to them like that

Re: GitHub Copilot Chat Leaked Prompt

#122

Post by runtime_blues » Sat, May 13, 2023, 3:33 AM UTC

jameshart wrote: post #109

Something that I find weird about these chat prompts (assuming they are real, not hallucinated): They're almost always written in second person*. "You are an AI programming assistant" "You are about to immerse yourself into the role of another Al model known as DAN" Who are these prompts addressed to? Who does the GPT think wrote them? The thing that confuses me is that these are text token prediction algorithms, und…

Early GPTs were fairly bad at following instructions. The innovation was RLHF, where human raters (Mechanical Turk style) would be asked to evaluate on how well the LLM is able to follow instructions stated as a part of the prompt, often in this style. Countless such ratings were incorporated into the training process itself.

So it did not happen out of the blue, and you didn't need a whole lot of existing webpages involving this sort of role play.

Re: GitHub Copilot Chat Leaked Prompt

#123

Post by simonw » Sat, May 13, 2023, 3:34 AM UTC

Here's why I don't think this leaked prompt is hallucinated (quoting from my tweets https://twitter.com/simonw/status/1657227047285166080 ):

Any time something like this happens a bunch of people suspect that it might be a hallucination, not the real prompt

I used to think that but I don't any more: prompt leaks are so easy to pull off, and I've not yet seen a documented case of a hallucinated but realistic leak

One of the reasons I no longer suspect hallucination is that the training cut-off date for OpenAI's LLMs - September 2021 - predates the point when this kind of prompt engineering became common enough that there would have been prompts like this in their training sets

Another reason is that we know that models give far greater weight to content in the prompt than content that's been trained into the model itself - that's why techniques like summarization and fact extraction from text work so well

Re: GitHub Copilot Chat Leaked Prompt

#124

Post by bqmjjx0kac » Sat, May 13, 2023, 3:36 AM UTC

simonw wrote: post #94

herpderperator wrote: post #92

Earlier quoted context omitted.

Is it actually able to base64 encode and decode completely correctly?

Yes, GPT-4 can at least.

At first I was shocked to hear this, but then I realized it's basically a substitution cipher.

Re: GitHub Copilot Chat Leaked Prompt

#125

Post by digdugdirk » Sat, May 13, 2023, 3:36 AM UTC

simonw wrote: post #103

Huh, this is the least interesting thing I've written about prompt injection in the last few weeks, but the only one to make it to the Hacker News homepage. Better recent posts: - Delimiters won’t save you from prompt injection - https://simonwillison.net/2023/May/11/delimiters-wont-save-y... - talks about why telling a model to follow delimiters like ``` won't protect against prompt injection, despite that being men…

Thanks! Love your writing. One question for you - how do you absorb these new concepts and experiment with them so quickly? It seems like you have the output of a small team, not just one person.

Re: GitHub Copilot Chat Leaked Prompt

#126

Post by nborwankar » Sat, May 13, 2023, 3:37 AM UTC

skissane wrote: post #54

naet wrote: post #19

I think that a lot of the limits placed on these models / chat services don't do much to remove underlying bias but rather attempt to obfuscate them from the general public. ChatGPT, Dall-e, etc all make assumptions about identity or politics but try to sidestep direct requests around those topics to appear more neutral... but the bias still exists in the model and affects the answers.

> ChatGPT, Dall-e, etc all make assumptions about identity or politics but try to sidestep direct requests around those topics to appear more neutral... but the bias still exists in the model and affects the answers. In the case of ChatGPT, I’d love to know how much of the bias is in the original (pre)training data, and how much is due to OpenAI’s human trainers. It is so careful to avoid every bias which is condemne…

IMHO it may be more accurate to say “the US has more fluent speakers of American English than any other country in the world.”

Re: GitHub Copilot Chat Leaked Prompt

#127

Post by cloudking » Sat, May 13, 2023, 3:38 AM UTC

I think one solution to grounding models with prompts is to have a secondary model that does the grounding based on the output of the primary model. Essentially run the output from the primary model to the secondary model, have it apply the prompt rules, reformat the response and send it back. The communication between primary and secondary model should not be exposed to the internet, like having a public web server and private database server.

Re: GitHub Copilot Chat Leaked Prompt

#128

Post by O5vYtytb » Sat, May 13, 2023, 3:39 AM UTC

jsheard wrote: post #29

Wowfunhappy wrote: post #15

With this and similar findings, how do we know the prompt is genuine and not a hallucination?

The source mentions in a reply that they were able to reproduce this exact text multiple times through different prompt injection approaches, right down to the typo "you name" rather than "your name" , which seems unlikely to happen if it were making it up out of thin air. I wonder if "you name" is a load bearing typo that breaks something else if corrected, so they left it in on purpose.

> right down to the typo "you name" rather than "your name", which seems unlikely to happen if it were making it up out of thin air.

Why is it unlikely? Why does prompting it different ways and getting the same result make it unlikely?

Re: GitHub Copilot Chat Leaked Prompt

#129

Post by simonw » Sat, May 13, 2023, 3:40 AM UTC

digdugdirk wrote: post #125

simonw wrote: post #103

Huh, this is the least interesting thing I've written about prompt injection in the last few weeks, but the only one to make it to the Hacker News homepage. Better recent posts: - Delimiters won’t save you from prompt injection - https://simonwillison.net/2023/May/11/delimiters-wont-save-y... - talks about why telling a model to follow delimiters like ``` won't protect against prompt injection, despite that being men…

Thanks! Love your writing. One question for you - how do you absorb these new concepts and experiment with them so quickly? It seems like you have the output of a small team, not just one person.

I'm "self-employed" aka I don't have anyone to tell me what else to spend my time on!

Re: GitHub Copilot Chat Leaked Prompt

#130

Post by wmidwestranger » Sat, May 13, 2023, 3:40 AM UTC

jameshart wrote: post #109

Something that I find weird about these chat prompts (assuming they are real, not hallucinated): They're almost always written in second person*. "You are an AI programming assistant" "You are about to immerse yourself into the role of another Al model known as DAN" Who are these prompts addressed to? Who does the GPT think wrote them? The thing that confuses me is that these are text token prediction algorithms, und…

> ... these are text token prediction algorithms, underneath.

If I had to take a wild guess, my guess would be that the prediction probabilities are very dependent on context, so by changing the context, the entire slate of probabilities shift.