Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

GitHub Copilot Chat Leaked Prompt

twitter.com

Post reply on HN
Print view View original

261–270 of 628 posts

Re: GitHub Copilot Chat Leaked Prompt

#261

Post by elcomet »

stevenhuang wrote: post #181
alex_sf wrote: post #172

Earlier quoted context omitted.

It's not being pedantic. RLHF and instruction tuning are completely different things. Painting with watercolors does not make water paint. Nearly all popular local models are instruction tuned, but are not RLHF'd. The OAI GPT series are not the only LLMs in the world.

Man it really doesn't need to be said that RLHF is not the only way to instruct tune. The point of my comment was to say that was how GPT3.5 was instruct tuned, via RLHF through a question answer dataset. At least we have this needless nerd snipe so others won't be potentially misled by my careless quip.

But that's still false. RLHF is not instruction fine-tuning. It is alignment. GPT 3.5 was first fine-tuned (supervised, not RL) on an instruction dataset, and then aligned to human expectations using RLHF.

Re: GitHub Copilot Chat Leaked Prompt

#262

Post by scotty79 »

jameshart wrote: post #109

Something that I find weird about these chat prompts (assuming they are real, not hallucinated): They're almost always written in second person*. "You are an AI programming assistant" "You are about to immerse yourself into the role of another Al model known as DAN" Who are these prompts addressed to? Who does the GPT think wrote them? The thing that confuses me is that these are text token prediction algorithms, und…

I think it's written that way because those models were trained on a lot of dialogs where one person uses second person form to command, and the other responds. Such material was used specifically to make this kind of dialog to work with users. See alpaca vs llama.

Re: GitHub Copilot Chat Leaked Prompt

#263

Post by scotty79 »

jameshart wrote: post #149
tel wrote: post #133

Earlier quoted context omitted.

If you play with a "raw" model such as LLaMA you'll find what you suggest is true. These models do what you'd expect of a model that was trained to predict the next token. It's quite tricky to convince such a model to do what you want. You have to conceptualize it and then imagine an optimal prefix leading to the sort of output you've conceptualized. That said, people discovered some fairly general-purpose prefixes,…

Right. But who's the 'you' who's being addressed by the {:system} prompt? Who is the {:assistant} supposed to think the {:system} is? Why should the {:assistant} output tokens that make it do what the {:system} tells it to? After all, the {:user} doesn't. The {:system} doesn't provide any instructions for how the {:user} is supposed to behave, the {:user} tokens are chosen arbitrarily and don't match the probabilitie…

> This all just seems like an existential nightmare.

I think using your native language just messes with your brain. When you hear "you" you think there someone being directly addressed. While this is just a word like "Você" that is used just to cause the artificial neural network trained on words to respond in prefered way.

Re: GitHub Copilot Chat Leaked Prompt

#264

Post by shzhdbi09gv8ioi »

ImHereToVote wrote: post #252
otikik wrote: post #227

Earlier quoted context omitted.

You are anthropomorphing. The machine doesn’t “really” understand, it’s just “simulating” it understands. “You” is “3 characters on an input string that are used to configure a program”. The prompt could have been any other thing, including a binary blob. It’s just more convenient for humans to use natural language to communicate, and the machine already has natural language features, so they used that instead of cre…

How do you know you aren't just "simulating" understanding?

[flagged]

Re: GitHub Copilot Chat Leaked Prompt

#265

Post by Lewton »

famouswaffles wrote: post #99
zarzavat wrote: post #63

Earlier quoted context omitted.

It’s possible but I feel that if an LLM flips styles, it will stick to that style afterwards. And the more advanced LLMs (I could be wrong but iirc Copilot chat is supposed to be GPT-4?) are much less likely to flip styles in the middle. Bigger models tend to be more coherent. I don’t think the Turing test has been passed by current SOTA LLMs, AI generated text still feels “off”, formulaic and flat, it doesn’t have t…

Current LLMs are deliberately trained to have a "flat, kind of robotic" default voice. Passing the Turing Test is not for a lack of ability here.

As long as prompt injection is possible, there’s zero LLMs that pass the Turing test

Re: GitHub Copilot Chat Leaked Prompt

#267

Post by rgaino »

jonplackett wrote: post #259

This is fascinating in so many ways. First - it is SO LONG - 500 tokens before any actual content. That’s a fairly hefty chunk of $ with GPT-4 to have to include with every single request. Second it’s interesting just how many times they have to tell it not to be offensive and argumentative. Third it’s hilarious just how easily it have up the secrets when it thinks the guy is from OpenAI. Getting GPT to stay on-task…

I came here to ask about the cost of this based on the number of tokens. Since this prompt is repeated on every single request, isn't there a way to embed it when you load the model? Or they simply use the raw OpenAI API like any of us mortals?

Re: GitHub Copilot Chat Leaked Prompt

#268

Post by robga »

otikik wrote: post #227
jameshart wrote: post #149

Earlier quoted context omitted.

Right. But who's the 'you' who's being addressed by the {:system} prompt? Who is the {:assistant} supposed to think the {:system} is? Why should the {:assistant} output tokens that make it do what the {:system} tells it to? After all, the {:user} doesn't. The {:system} doesn't provide any instructions for how the {:user} is supposed to behave, the {:user} tokens are chosen arbitrarily and don't match the probabilitie…

You are anthropomorphing. The machine doesn’t “really” understand, it’s just “simulating” it understands. “You” is “3 characters on an input string that are used to configure a program”. The prompt could have been any other thing, including a binary blob. It’s just more convenient for humans to use natural language to communicate, and the machine already has natural language features, so they used that instead of cre…

> You are anthropomorphing.

Agreed. The situation is so alien that we are prone to attribute human like terms to describe it.

> The machine doesn’t “really” understand, it’s just “simulating” it understands.

You are actually displaying a subtle form of anthropomorphism with this statement. You're comparing a human-like quality (“understands”) with the AI.

Your point still stands and your final para is well said - but it shows the difficult nature of discourse around the topic.

Re: GitHub Copilot Chat Leaked Prompt

#269

Post by afro88 »

bskap wrote: post #162
afro88 wrote: post #25

Earlier quoted context omitted.

But also, how would it even know if the code is copyrighted?

There's four ways for code to not be copyrighted (in the US): 1. The author died more than 70 years ago or it was owned by a corporation and it's been 95 years since publication 2. It was written prior to 1989 and did not include a copyright notice. 3. It was written by the government 4. The author explicitly released it into the public domain 1 and 2 probably don't cover much code on the Internet. So unless it's a g…

But how does an LLM actually know and enforce this? It can be basically tricked into anything

Re: GitHub Copilot Chat Leaked Prompt

#270

Post by afro88 »

pabs3 wrote: post #148
afro88 wrote: post #25

Earlier quoted context omitted.

But also, how would it even know if the code is copyrighted?

Code is copyrighted by default according to the law, very little code is actually public domain.

Understood, but how does an LLM actually know that the code it's spitting out already exists somewhere else that isn't public domain?
Post reply on HN