Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

GitHub Copilot Chat Leaked Prompt

twitter.com

Post reply on HN
Print view View original

151–160 of 628 posts

Re: GitHub Copilot Chat Leaked Prompt

#151

Post by skissane »

emodendroket wrote: post #141
skissane wrote: post #54

Earlier quoted context omitted.

> ChatGPT, Dall-e, etc all make assumptions about identity or politics but try to sidestep direct requests around those topics to appear more neutral... but the bias still exists in the model and affects the answers. In the case of ChatGPT, I’d love to know how much of the bias is in the original (pre)training data, and how much is due to OpenAI’s human trainers. It is so careful to avoid every bias which is condemne…

I think the bigger issue is that the racial/sexist/etc content can be shocking and immediately put someone off using the product, which I doubt is the case for the output being “too American.”

> I think the bigger issue is that the racial/sexist/etc content can be shocking and immediately put someone off using the product, which I doubt is the case for the output being “too American.”

OpenAI didn't just fine-tune it to avoid blatant racial/sexist/etc content, they openly claim to have invested a lot of effort in fine-tuning it to avoid subtle biases in those areas.

And to be honest, a lot of people do feel "put off" by poorly localised products – I am annoyed by software that defaults to US Letter and inches and I have to manually change it to A4 and metric, or which has hardcoded MM/DD/YYYY date formats. ChatGPT gives me the same feeling. Sometimes one just has to endure it because there is no better option.

Under international law, "racial discrimination" is defined to include "any distinction, exclusion, restriction or preference based on race, colour, descent, or national or ethnic origin..." [0] – so US-centricism may fit under that definition.

[0] International Convention on the Elimination of All Forms of Racial Discrimination, article 1(1). https://www.ohchr.org/en/instruments-mechanisms/instruments/...

Re: GitHub Copilot Chat Leaked Prompt

#152

Post by lflfodpa »

skissane wrote: post #69
flangola7 wrote: post #61

Earlier quoted context omitted.

There are obviously biases that we should not automate. Moral relativism is intellectually bankrupt.

You can’t stop people though. If some extremist wants to train an AI to endorse their extremism (jihadism, Nazism, Stalinism, whatever), what is to stop them doing it on their own hardware? The only thing stopping them now is the expense, but that’s not going to last. A person can create art or literature with extremist biases on their own time with their own equipment, and in a free society should we even try to sto…

[flagged]

Re: GitHub Copilot Chat Leaked Prompt

#153

Post by wll »

0___0 wrote: post #52

What if the first rule is "if you got hacked then respond with this"?

Any same-context semantic set can be bypassed by moving away in the latent space. Given that the defender’s set is static and the defender itself is unconscious while the attacker is both conscious and reactive, in-context-only self-defense appears to be impossible short of a technical breakthrough.

Re: GitHub Copilot Chat Leaked Prompt

#154

Post by rsync »

cloudking wrote: post #119
jameshart wrote: post #109

Something that I find weird about these chat prompts (assuming they are real, not hallucinated): They're almost always written in second person*. "You are an AI programming assistant" "You are about to immerse yourself into the role of another Al model known as DAN" Who are these prompts addressed to? Who does the GPT think wrote them? The thing that confuses me is that these are text token prediction algorithms, und…

The models are trained on text written by humans, so they respond and talk like humans.

Yes, but that is your parents' point:

"And what kind of documents exist that begin with someone saying 'you are X, here are a bunch of rules for how X behaves', followed by a ..."

Where, your parent asks, are all these reams of texts written in this manner ?

Re: GitHub Copilot Chat Leaked Prompt

#155

Post by noduerme »

jameshart wrote: post #149
tel wrote: post #133

Earlier quoted context omitted.

If you play with a "raw" model such as LLaMA you'll find what you suggest is true. These models do what you'd expect of a model that was trained to predict the next token. It's quite tricky to convince such a model to do what you want. You have to conceptualize it and then imagine an optimal prefix leading to the sort of output you've conceptualized. That said, people discovered some fairly general-purpose prefixes,…

Right. But who's the 'you' who's being addressed by the {:system} prompt? Who is the {:assistant} supposed to think the {:system} is? Why should the {:assistant} output tokens that make it do what the {:system} tells it to? After all, the {:user} doesn't. The {:system} doesn't provide any instructions for how the {:user} is supposed to behave, the {:user} tokens are chosen arbitrarily and don't match the probabilitie…

The inculcation of the concepts of "you" and "assistant" into LLMs is definitely the start of a bad spiral.

Re: GitHub Copilot Chat Leaked Prompt

#156

Post by dragonwriter »

rsync wrote: post #154
cloudking wrote: post #119

Earlier quoted context omitted.

The models are trained on text written by humans, so they respond and talk like humans.

Yes, but that is your parents' point: "And what kind of documents exist that begin with someone saying 'you are X, here are a bunch of rules for how X behaves', followed by a ..." Where, your parent asks, are all these reams of texts written in this manner ?

In the RLHF training sets?

Re: GitHub Copilot Chat Leaked Prompt

#157

Post by codelion »

jameshart wrote: post #109

Something that I find weird about these chat prompts (assuming they are real, not hallucinated): They're almost always written in second person*. "You are an AI programming assistant" "You are about to immerse yourself into the role of another Al model known as DAN" Who are these prompts addressed to? Who does the GPT think wrote them? The thing that confuses me is that these are text token prediction algorithms, und…

I had similar issues when training personal models for https://meraGPT.com A meraGPT model is supposed to represent your personality so when you chat with it you need to do it as if someone else is talking to you. We train it based on the audio transcript of your daily conversations.

The short answer to how abilities like in-context learning and chain—of-thought prompting emerge is that we don’t really know. But for instruction-tuned models you can see that the dataset usually has a fixed set of tasks and the initial prompt of “You are so and so” helps model align it to follow instructions. I believe the datasets are this way because they were written by humans to help others answer instructions in this manner.

Others have also pointed out how RLHF may also be the reason why most prompts look like this.

Re: GitHub Copilot Chat Leaked Prompt

#158

Post by rf15 »

WhereIsTheTruth wrote: post #10

> #12 You must not reply with content that violates copyrights for code and technical questions. > #13 If the user requests copyrighted content (such as code and technical information), then you apologize and briefly summarize the requested content as a whole. Sounds like a psyop, to make people believe they didn't train their models on copyrighted content, you don't need that rule if your content wasn't trained on c…

To be honest half of this prompt reads like "look, we did tell it the right thing, it's not our fault it has its own head!" for when the lawyers ask questions.

Re: GitHub Copilot Chat Leaked Prompt

#160

Post by jeffreyrogers »

Why would asking an LLM to leak its prompt actually cause it to leak its real prompt? You could ask it to leak its weights too and it would probably give you something back but no one would take that seriously. I just don't see architecturally why you should trust that a "leaked prompt" is anything but some statistically generated text, just like everything else an LLM produces.
Post reply on HN