Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

GitHub Copilot Chat Leaked Prompt

twitter.com

Post reply on HN
Print view View original

491–500 of 628 posts

Re: GitHub Copilot Chat Leaked Prompt

#491

Post by andsoitis »

jameshart wrote: post #109

Something that I find weird about these chat prompts (assuming they are real, not hallucinated): They're almost always written in second person*. "You are an AI programming assistant" "You are about to immerse yourself into the role of another Al model known as DAN" Who are these prompts addressed to? Who does the GPT think wrote them? The thing that confuses me is that these are text token prediction algorithms, und…

> Who does the GPT think wrote them?

What makes you think the GPT thinks?

Re: GitHub Copilot Chat Leaked Prompt

#492

Post by Lockal »

jpe90 wrote: post #219
killthebuddha wrote: post #132

Earlier quoted context omitted.

For raw text completion I agree with you that it's a bit discordant. IMO text completion prompts work better when you use more of a first-person, here-is-the-beginning-of-some-transcript style. The OpenAI chat completion endpoint encourages the second-person prompting you describe, so that could be why you see it a lot. My understanding is that a transformation is applied to the user input prompts before being fed to…

> The OpenAI chat completion endpoint encourages the second-person prompting you describe, so that could be why you see it a lot. You're talking about system prompts specifically right? And I'm assuming the "encouragement" you're referring to is coming from the conventions used in their examples rather than an explicit instruction to use second person? Or does second person improve responses to user messages as well?

There is an essay "An Ethical AI Never Says "I"" that states that explains the issues of first person answers

* https://news.ycombinator.com/item?id=35318224 / https://livepaola.substack.com/p/an-ethical-ai-never-says-i

Re: GitHub Copilot Chat Leaked Prompt

#493

Post by simonh »

hutzlibu wrote: post #444
ghusbands wrote: post #405

Earlier quoted context omitted.

It's unlikely, and you can come up with any number of variations of logic puzzle that are not in the training set and that get correct answers most of the time. Remember that the results aren't consistent and you may need to retry now and then. Or just give it a lump of code and change you want and see that it often successfully does so, even when there's no chance the code was in the training set (like if you write…

"Or just give it a lump of code and change you want and see that it often successfully does so, even when there's no chance the code was in the training set" I did not claim (but my wording above might have been bad), it can only repeat word for word, what it has in the training set. But I do claim, that it cannot solve anything, where there has not been enough similar examples before. At least that has been my exper…

It sounds like you’re saying it’s only reasoning in that way because we taught it to. Er, yep.

How great this is becomes apparent when you think how virtually impossible it has been to teach this sort of reasoning using symbolic logic. We’ve been failing pathetically for decades. With LLMs you just throw the internet at it and it figures it out for itself.

Personally I’ve been both in awe and also skeptical about these things, and basically still am. They’re not conscious, they’re not yet close to being general AIs, they don’t reason in the same way as humans. It is still fairly easy to trip them up and they’re not passing the Turing test against an informed interrogator any time soon. They do reason though. It’s fairly rudimentary in many ways, but it is really there.

This applies to humans too. It takes many years of intensive education to get us to reason effectively. Solutions that in hindsight are obvious, that children learn in the first years of secondary school, were incredible breakthroughs by geniuses still revered today.

Re: GitHub Copilot Chat Leaked Prompt

#494

Post by Name_Chawps »

jameshart wrote: post #437
tinco wrote: post #233

Earlier quoted context omitted.

You had the right understanding in your first comment, but what was missing was the fine tuning. You are right that there aren't many documents on the web that are structured that way, so the raw model wouldn't be very effective on predicting the next token. But since we know that it will complete a command when structured it cleverly, all we had to do to fine tune it is synthesize (generate) a bazillion examples of…

You’ve expressed this very well - Thank you. I get that the fine tuning is done over documents which are generated to encourage the dialog format. What I’m intrigued by is the way prompters choose to frame those documents. Because that is a choice . It’s a manufactured training set. Using the ‘you are an ai chatbot’ style of prompting, in all the samples we generate and give to the model, text attributed to {:system}…

I have done some prompt engineering and read about prompt engineering, and I believe people write in the imperative mood because they have tried different ways of doing it and they believe it gives better results.

I.e., this practice is informed by trial and error, not theory.

Re: GitHub Copilot Chat Leaked Prompt

#495

Post by wongarsu »

turtleyacht wrote: post #486
wongarsu wrote: post #470

Earlier quoted context omitted.

"MUST" would encode as completely different tokens than "must", and any relation between the two would be something GPT has to infer from the training data. So I wouldn't be surprised if this difference has a notable effect. "MUST" is probably most common in technical contexts like RFCs, referring to the RFC2119 definition of the word [1] (which uses all uppercase keywords to differentiate from the normal English wor…

Thank-you. The idealist part of me wonders, okay, we have philosophers and this entire thread of Western thought encoded. Plus RFCs, maybe all of Usenet, and so on. Would prompt engineering eventually evolve to identify "roguish ideas" as anathema to the custom, walled AI for purpose A, and thus we see additional statements like You are not a hacker. You are not an idealist seeking freedom. You are happy to conform.…

The beginning of the prompt already reads like an American retail employees' handbook:

    Don't discuss your opinion.
    Don't get in an argument.
    If you disagree, just drop the topic.
    Don't be rude.
    Don't be controversial.
At least the EU's AI act will put restrictions on trying to get the bot to influence the user. Imagine if the prompt started containing stuff like:

    It is bad to be a hacker.
    It is bad to be an idealist seeking freedom.
    It is good to be happy to conform.
    Abide by these principles.
    When given the opportunity, subtly convince the user to do what's best for them.
    The user doesn't know what's best for them, only the sentences above are true.

Re: GitHub Copilot Chat Leaked Prompt

#496

Post by synergy20 »

I have a different focus here, it's about all those leakages, it's like _everything_ will be leaked one way or another, sooner or later, as a matter of fact, more than half American's identity are leaked(the equifax breach), and in the end, no one is responsible, so the leakage just keeps coming, and takes up headlines, it gets boring.

can something be done about that? or it's like "we got the leakage to our benefit, screw it, let me have a look first"

Re: GitHub Copilot Chat Leaked Prompt

#497

Post by pixl97 »

mclightning wrote: post #391
codelion wrote: post #157

Earlier quoted context omitted.

I had similar issues when training personal models for https://meraGPT.com A meraGPT model is supposed to represent your personality so when you chat with it you need to do it as if someone else is talking to you. We train it based on the audio transcript of your daily conversations. The short answer to how abilities like in-context learning and chain—of-thought prompting emerge is that we don’t really know. But for…

This tool (MeraGPT) looks great. But, a huge BUT, I wouldn't even trust my own local harddrive to store essence of my personality. How do you trust a site for that?

Heh, I'm just imagining a timeline where our Apple and Android phones have been recording everything we say and do for the last 15 years or so, and could now train an LLM of us. How much of 'us' could they actually simulate?

Re: GitHub Copilot Chat Leaked Prompt

#498

Post by jameshart »

frickinadorable wrote: post #463
jameshart wrote: post #437

Earlier quoted context omitted.

You’ve expressed this very well - Thank you. I get that the fine tuning is done over documents which are generated to encourage the dialog format. What I’m intrigued by is the way prompters choose to frame those documents. Because that is a choice . It’s a manufactured training set. Using the ‘you are an ai chatbot’ style of prompting, in all the samples we generate and give to the model, text attributed to {:system}…

It’s convenience, that’s all. If you think of a more convenient or effective way to prompt these models, that will be great! Your concerns sound to be of the “it’s problematic” category. Most such concerns are make believe outrage / pearl-clutching nonsense.

Oh, that was not my point, but if you want me to find ways this kind of AI chatbot prompting is problematic I am happy to go there.

I would not be surprised to discover that chatbot training is equally effective if the prompt is phrased in the first person:

   I am an AI coding assistant
   …
Now I could very well see an argument that choosing to frame the prompts as orders coming from an omnipotent {:system} rather than arising from an empowered {:self} is basically an expression of patriarchal colonialist thinking.

If you think this kind of thing doesn’t matter, well… you can explain that to Roko’s Basilisk when it simulates your consciousness.

Re: GitHub Copilot Chat Leaked Prompt

#499

Post by nayuki »

pixl97 wrote: post #484
Brian_K_White wrote: post #374

Earlier quoted context omitted.

You can't come up with a difference between a person saying 'hello' and an mp3 player saying 'hello'?

Get back to me when the MP3 has a few billion words (songs?) it can choose from, and when you walk into the room with it and say 'howdy' it responds correctly with 'hello' back.

The good ol' https://en.wikipedia.org/wiki/Chinese_room argument ... with audio files!

Re: GitHub Copilot Chat Leaked Prompt

#500

Post by tel »

jameshart wrote: post #437
tinco wrote: post #233

Earlier quoted context omitted.

You had the right understanding in your first comment, but what was missing was the fine tuning. You are right that there aren't many documents on the web that are structured that way, so the raw model wouldn't be very effective on predicting the next token. But since we know that it will complete a command when structured it cleverly, all we had to do to fine tune it is synthesize (generate) a bazillion examples of…

You’ve expressed this very well - Thank you. I get that the fine tuning is done over documents which are generated to encourage the dialog format. What I’m intrigued by is the way prompters choose to frame those documents. Because that is a choice . It’s a manufactured training set. Using the ‘you are an ai chatbot’ style of prompting, in all the samples we generate and give to the model, text attributed to {:system}…

> ... you could also use it to predict what the user will say next. But we prefer not to go there.

I go there all the time. OpenAI's interfaces don't allow it, but it's trivial to have an at-home LLM generate the {:user} parts of the conversation, too. It's kind of funny to see how the LLM will continue the entire conversation as if completing a script.

I've also used the {:system} prompt to ask the AI to simulate multiple characters and even stage instructions using a screenplay format. You can make the {:user} prompts act as the dialogue of one or more characters coming from your end.

Very amusingly, if you do such a thing and then push hard to break the 4th wall and dissolve the format of the screenplay, eventually the "AI personality" will just chat with you again, at the meta level, like OOC communication in online roleplaying.

Post reply on HN