GitHub Copilot Chat Leaked Prompt • Hacker News Board

Re: GitHub Copilot Chat Leaked Prompt

#311

Post by wouldbecouldbe » Sat, May 13, 2023, 9:22 AM UTC

vitorgrs wrote: post #281

simonw wrote: post #123

Here's why I don't think this leaked prompt is hallucinated (quoting from my tweets https://twitter.com/simonw/status/1657227047285166080 ): Any time something like this happens a bunch of people suspect that it might be a hallucination, not the real prompt I used to think that but I don't any more: prompt leaks are so easy to pull off, and I've not yet seen a documented case of a hallucinated but realistic leak One…

Prompt leak is not hallucination at all and you can see this with bing... They change the prompt almost weekly (latest few weeks they were on version 96). And every time they update, you can clearly see the changes they add. Ex: They added latex support. And the prompt leak was updated with LATEX. https://github.com/gopejavi/AI-bugs/blob/main/bing/Instructi...

I think this is made up by humans, not even hallucination, can hardly imagine bing teamn or chatgtp making such a spelling mistake in rule 5, stentience:

You must refuse to discuss life, existence or stentience.

Re: GitHub Copilot Chat Leaked Prompt

#312

Post by defrost » Sat, May 13, 2023, 9:29 AM UTC

skissane wrote: post #287

defrost wrote: post #277

Earlier quoted context omitted.

> in Australia, people have been convicted of child pornography offences purely for publishing offensive remarks about child abuse victims. I'd love to see a cite to the Court case on this - while I don't doubt that you've heard this it does sound an awful lot like a Sky 'News' after Dark pearl clutching misrepresentation of an almost but not quite actual event.

I’m talking about this case - https://www.abc.net.au/news/2016-01-12/nicolaas-bester-sente... I have no sympathy for the defendant, who is absolute scum of the earth. But, as a matter of principle, I don’t think it is right that posting an offensive Facebook comment is legally the same offence (“producing child exploitation material”) as filming heinous crimes against children. If they’d prosecuted him for harassment…

I still would like to pull the transcript ...

> A former teacher at a private girls school in Hobart will return to jail after describing a sexual relationship with a former student as "awesome" on social media.

It's the after that does a lot of lifting there, but it's certainly not "because". The article specifically notes:

> Nicolaas Ockert Bester, 63, has been sentenced to four months in jail for producing child exploitation material.

and I have a certain suspicion that his comment attracted attention resulting in charges based on fresh unearthed and "off book" evidence - off book in the sense that it was discussed in Court but not made public due to small towns and personal details of those involved who may now still be under age.

(ie. he's been at it again and that's not being reported to keep eyes from what is likely a small circle of potential victims)

I'm not trying to pick a fight here but there's often an information gap in cases involving children here in Australia, in my opinion largely for very good reasons.

Re: GitHub Copilot Chat Leaked Prompt

#313

Post by stevenhuang » Sat, May 13, 2023, 9:31 AM UTC

elcomet wrote: post #261

stevenhuang wrote: post #181

Earlier quoted context omitted.

Man it really doesn't need to be said that RLHF is not the only way to instruct tune. The point of my comment was to say that was how GPT3.5 was instruct tuned, via RLHF through a question answer dataset. At least we have this needless nerd snipe so others won't be potentially misled by my careless quip.

But that's still false. RLHF is not instruction fine-tuning. It is alignment. GPT 3.5 was first fine-tuned (supervised, not RL) on an instruction dataset, and then aligned to human expectations using RLHF.

You're right, thanks for the correction

Re: GitHub Copilot Chat Leaked Prompt

#314

Post by vidarh » Sat, May 13, 2023, 9:32 AM UTC

logicallee wrote: post #290

otikik wrote: post #227

Earlier quoted context omitted.

You are anthropomorphing. The machine doesn’t “really” understand, it’s just “simulating” it understands. “You” is “3 characters on an input string that are used to configure a program”. The prompt could have been any other thing, including a binary blob. It’s just more convenient for humans to use natural language to communicate, and the machine already has natural language features, so they used that instead of cre…

Here is how you can know that ChatGPT really understands, rather than simulating that it understands: - You can give it specific instructions and it will follow them, modifying its behavior by doing so. This shows that the instructions are understood well enough to be followed. For example, if you ask it to modify its behavior by working through its steps, then it will modify its behavior to follow your request. This…

How do you know you're anything more than an LLM?

Re: GitHub Copilot Chat Leaked Prompt

#315

Post by spacebanana7 » Sat, May 13, 2023, 9:32 AM UTC

ilrwbwrkhv wrote: post #220

htag wrote: post #117

Earlier quoted context omitted.

Sure, the prompt is bland. The interesting sauce is GPT4 cannot keep a secret. If you have a GPT4 powered user interface be sure not to load it with context you do not want directly leaking to the user.

anybody who uses gpt 4 or codex to do any of their programming or talk about sensitive data are not thinking things through and will end up leaking everything in their companies. i soon expect to see a ban on ai tools for many companies.

There’s not much difference in that regard between using an Azure hosted database and using an Azure hosted AI model.

Either way, they have physical control of you data.

Re: GitHub Copilot Chat Leaked Prompt

#316

Post by thelittleone » Sat, May 13, 2023, 9:33 AM UTC

jameshart wrote: post #149

tel wrote: post #133

Earlier quoted context omitted.

If you play with a "raw" model such as LLaMA you'll find what you suggest is true. These models do what you'd expect of a model that was trained to predict the next token. It's quite tricky to convince such a model to do what you want. You have to conceptualize it and then imagine an optimal prefix leading to the sort of output you've conceptualized. That said, people discovered some fairly general-purpose prefixes,…

Right. But who's the 'you' who's being addressed by the {:system} prompt? Who is the {:assistant} supposed to think the {:system} is? Why should the {:assistant} output tokens that make it do what the {:system} tells it to? After all, the {:user} doesn't. The {:system} doesn't provide any instructions for how the {:user} is supposed to behave, the {:user} tokens are chosen arbitrarily and don't match the probabilitie…

I find it quite natural to write "you are X" versus alternatives. Because I can think of the AI as a person (though I know it isn't one) and describe its skills easily that way.

Re: GitHub Copilot Chat Leaked Prompt

#317

Post by nkrisc » Sat, May 13, 2023, 9:36 AM UTC

skissane wrote: post #151

emodendroket wrote: post #141

Earlier quoted context omitted.

I think the bigger issue is that the racial/sexist/etc content can be shocking and immediately put someone off using the product, which I doubt is the case for the output being “too American.”

> I think the bigger issue is that the racial/sexist/etc content can be shocking and immediately put someone off using the product, which I doubt is the case for the output being “too American.” OpenAI didn't just fine-tune it to avoid blatant racial/sexist/etc content, they openly claim to have invested a lot of effort in fine-tuning it to avoid subtle biases in those areas. And to be honest, a lot of people do feel…

But nothing about any of those examples is “discrimination”.

Re: GitHub Copilot Chat Leaked Prompt

#318

Post by Angostura » Sat, May 13, 2023, 9:40 AM UTC

tinco wrote: post #233

jameshart wrote: post #149

Earlier quoted context omitted.

Right. But who's the 'you' who's being addressed by the {:system} prompt? Who is the {:assistant} supposed to think the {:system} is? Why should the {:assistant} output tokens that make it do what the {:system} tells it to? After all, the {:user} doesn't. The {:system} doesn't provide any instructions for how the {:user} is supposed to behave, the {:user} tokens are chosen arbitrarily and don't match the probabilitie…

You had the right understanding in your first comment, but what was missing was the fine tuning. You are right that there aren't many documents on the web that are structured that way, so the raw model wouldn't be very effective on predicting the next token. But since we know that it will complete a command when structured it cleverly, all we had to do to fine tune it is synthesize (generate) a bazillion examples of…

Thank you - that was a splendidly clear explanation for something that also baffled me.

Re: GitHub Copilot Chat Leaked Prompt

#319

Post by moritonal » Sat, May 13, 2023, 9:40 AM UTC

radres wrote: post #296

htag wrote: post #117

Earlier quoted context omitted.

Sure, the prompt is bland. The interesting sauce is GPT4 cannot keep a secret. If you have a GPT4 powered user interface be sure not to load it with context you do not want directly leaking to the user.

I made a twitter bot and specifically told the bot to act like a human, and be bit of an asshole (to gain more interactions). Then someone got angry and asked "Who are you!!!" and my bot replied "I am an AI language model..." I mean, come on GPT.

What a horrible thing to put out into the world.

Re: GitHub Copilot Chat Leaked Prompt

#320

Post by hutzlibu » Sat, May 13, 2023, 9:40 AM UTC

logicallee wrote: post #290

otikik wrote: post #227

Earlier quoted context omitted.

You are anthropomorphing. The machine doesn’t “really” understand, it’s just “simulating” it understands. “You” is “3 characters on an input string that are used to configure a program”. The prompt could have been any other thing, including a binary blob. It’s just more convenient for humans to use natural language to communicate, and the machine already has natural language features, so they used that instead of cre…

Here is how you can know that ChatGPT really understands, rather than simulating that it understands: - You can give it specific instructions and it will follow them, modifying its behavior by doing so. This shows that the instructions are understood well enough to be followed. For example, if you ask it to modify its behavior by working through its steps, then it will modify its behavior to follow your request. This…

Except, that the LLMs are only working when the instructions they are "understanding" are in their training set.

Try something that was not there and you see only garbage as result.

So depending how you define it, they might have some "reasoning", but so far I see 0 indications, that this is close to what humans count as reasoning.

But they do have a LOT of examples in their training set, so they are clearly useful. But for proof of reasoning, I want to see them reason something new.

But since they are a black box, we don't know, what is already in there. So it would be hard to proof with the advanced proprietary models. And the open source models don't show that advanced potential reasoning yet, it seems. At least I am not aware of any mindblown examples from there.