Maybe I'm missing something, but how do we know that this is actually a leaked prompt?
GitHub Copilot Chat Leaked Prompt
131–140 of 628 posts
Re: GitHub Copilot Chat Leaked Prompt
#132Something that I find weird about these chat prompts (assuming they are real, not hallucinated): They're almost always written in second person*. "You are an AI programming assistant" "You are about to immerse yourself into the role of another Al model known as DAN" Who are these prompts addressed to? Who does the GPT think wrote them? The thing that confuses me is that these are text token prediction algorithms, und…
For raw text completion I agree with you that it's a bit discordant. IMO text completion prompts work better when you use more of a first-person, here-is-the-beginning-of-some-transcript style.
The OpenAI chat completion endpoint encourages the second-person prompting you describe, so that could be why you see it a lot. My understanding is that a transformation is applied to the user input prompts before being fed to the underlying model, so it's possible that the model receives a more natural transcription-style prompt.
You might be interested in this paper, which explores ways to help non-experts write prompts https://dl.acm.org/doi/abs/10.1145/3544548.3581388.
Re: GitHub Copilot Chat Leaked Prompt
#133Something that I find weird about these chat prompts (assuming they are real, not hallucinated): They're almost always written in second person*. "You are an AI programming assistant" "You are about to immerse yourself into the role of another Al model known as DAN" Who are these prompts addressed to? Who does the GPT think wrote them? The thing that confuses me is that these are text token prediction algorithms, und…
If you play with a "raw" model such as LLaMA you'll find what you suggest is true. These models do what you'd expect of a model that was trained to predict the next token.
It's quite tricky to convince such a model to do what you want. You have to conceptualize it and then imagine an optimal prefix leading to the sort of output you've conceptualized. That said, people discovered some fairly general-purpose prefixes, e.g.
Q: What is the 3rd law of Thermodynamics?
A:
This ends up being way more convenient. Now all the prompter has to do is conceptualize what they want and expect that the LLM will receive it as instruction. It simplifies prompting and makes the LLM more steerable, more useful, more helpful.
This is further refined through the use of explicit {:user}, {:assistant}, and {:system} tags which divide LLM contexts into different segments with explicit interpretations of the meaning of each segment. This is where "chat instruction" arises in models such as GPT-3.5.
Re: GitHub Copilot Chat Leaked Prompt
#134Maybe I'm missing something, but how do we know that this is actually a leaked prompt?
We don't for sure, but I'm pretty confident it is. Here's why: https://news.ycombinator.com/item?id=35925239
Re: GitHub Copilot Chat Leaked Prompt
#135Something that I find weird about these chat prompts (assuming they are real, not hallucinated): They're almost always written in second person*. "You are an AI programming assistant" "You are about to immerse yourself into the role of another Al model known as DAN" Who are these prompts addressed to? Who does the GPT think wrote them? The thing that confuses me is that these are text token prediction algorithms, und…
Responding to prompts like that are part of the 'instruction tuning' process. After an LLM is trained on a large dataset, it will do a decent job of completion, which acts like you describe.
The next step is to further tune it with a specific format. You'll feed in examples like so:
SystemPrompt: You are a rude AI.
User: Hello there!
Assistant: You're lame, go away.
SystemPrompt: You are a pleasant AI.
User: Hello there!
Assistant: Hello, friend!
SystemPrompt: You are a pleasant AI.
User: [user prompt]
Assistant:
Additional tuning (RLHF, etc.) is orthogonal.
Re: GitHub Copilot Chat Leaked Prompt
#136Earlier quoted context omitted.
The source mentions in a reply that they were able to reproduce this exact text multiple times through different prompt injection approaches, right down to the typo "you name" rather than "your name" , which seems unlikely to happen if it were making it up out of thin air. I wonder if "you name" is a load bearing typo that breaks something else if corrected, so they left it in on purpose.
> right down to the typo "you name" rather than "your name", which seems unlikely to happen if it were making it up out of thin air. Why is it unlikely? Why does prompting it different ways and getting the same result make it unlikely?
It's also very unlikely that an LLM would hallucinate a prompt with a spelling mistake in it. LLMs are really good at spelling.
Re: GitHub Copilot Chat Leaked Prompt
#137How do we know these are real rules or were just made up on the spot of what it thinks the rules should be for some hypothetic AI engine?
Re: GitHub Copilot Chat Leaked Prompt
#138I feel like we've put too much emphasis on the prompts, as though they're some sort of special sauce. In reality, though, they're all pretty bland. It's like getting ahold of an employee handbook for Applebees. It feels scandalous to see the inner workings, because we're not supposed to see it, but ultimately it's basically what you would have guessed anyway.
Sure, the prompt is bland. The interesting sauce is GPT4 cannot keep a secret. If you have a GPT4 powered user interface be sure not to load it with context you do not want directly leaking to the user.
[deleted]
Re: GitHub Copilot Chat Leaked Prompt
#139Huh, this is the least interesting thing I've written about prompt injection in the last few weeks, but the only one to make it to the Hacker News homepage. Better recent posts: - Delimiters won’t save you from prompt injection - https://simonwillison.net/2023/May/11/delimiters-wont-save-y... - talks about why telling a model to follow delimiters like ``` won't protect against prompt injection, despite that being men…
Thanks! Love your writing. One question for you - how do you absorb these new concepts and experiment with them so quickly? It seems like you have the output of a small team, not just one person.
you don't really know how new that is for different people. I'd imagine 3 months of learning about something interesting gives you quite a good idea about the topic, at least good enough to write about it.
Re: GitHub Copilot Chat Leaked Prompt
#140Something that I find weird about these chat prompts (assuming they are real, not hallucinated): They're almost always written in second person*. "You are an AI programming assistant" "You are about to immerse yourself into the role of another Al model known as DAN" Who are these prompts addressed to? Who does the GPT think wrote them? The thing that confuses me is that these are text token prediction algorithms, und…
They have been RLHF (reinforcement learning with human feedback) tuned.
In essence they've been fine tuned to be able to follow instructions.