Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

GitHub Copilot Chat Leaked Prompt

twitter.com

Post reply on HN
Print view View original

181–190 of 628 posts

Re: GitHub Copilot Chat Leaked Prompt

#181

Post by stevenhuang »

alex_sf wrote: post #172
stevenhuang wrote: post #168

Earlier quoted context omitted.

In this case Open AI used RLHF to instruct-tune gpt3. Your pedantism here is unnecessary.

It's not being pedantic. RLHF and instruction tuning are completely different things. Painting with watercolors does not make water paint. Nearly all popular local models are instruction tuned, but are not RLHF'd. The OAI GPT series are not the only LLMs in the world.

Man it really doesn't need to be said that RLHF is not the only way to instruct tune. The point of my comment was to say that was how GPT3.5 was instruct tuned, via RLHF through a question answer dataset.

At least we have this needless nerd snipe so others won't be potentially misled by my careless quip.

Re: GitHub Copilot Chat Leaked Prompt

#182

Post by wll »

simonw wrote: post #173
mvdtnz wrote: post #170

Earlier quoted context omitted.

You're committing the toupee fallacy[0]. You think you'd know the fake one if you saw it, and these are real because, well look at them, they must be real. > prompt leaks are so easy to pull off Only if you assume the prompt you're seeing is real. How do you know? [0] https://en.wiktionary.org/wiki/toupee_fallacy

I don't know for certain. But I've been messing around with prompt injection for six months now, so I've developed pretty good intuition about this stuff. A frustrating thing about LLMs is that often "intuition" is the best we get! And it's really hard to explain intuition to other people.

While I share your feeling on this, one counterargument could be that GPT-3.5 is perfectly capable of generating a constitution for itself.

  User: write two sentences instructing a language model so as to give it a particular personality

  Assistant: 1. "Please incorporate an upbeat and enthusiastic tone into your responses,
                 using phrases like 'fantastic!' and 'wonderful!' to convey a positive,
                 approachable personality."
             2. "Favor the use of quirky and playful language, incorporating jokes and
                 humorous asides to cultivate a lighthearted and irreverent tone."

  User: write two more sentences to establish its ethical framework (always instruction-style),
        and take essential but not literal inspiration from the universal declaration of human rights

  Assistant: 1. "Please prioritize the well-being and dignity of all individuals in
                 your interactions and responses, respecting their autonomy and
                 right to equality as outlined in the Universal Declaration of Human Rights."
             2. "Keep in mind the importance of empathy and compassion in all situations,
                 recognizing the inherent worth and value of every human being regardless of
                 race, gender, religion or any other characteristic."
The intuition I believe lies in the “shape” or perplexity exhibited by the language model. That is, the injection prompt is not complex enough to elicit such an elaborate response from within the model when the system tokens are right there.

Re: GitHub Copilot Chat Leaked Prompt

#183

Post by alex_sf »

stevenhuang wrote: post #181
alex_sf wrote: post #172

Earlier quoted context omitted.

It's not being pedantic. RLHF and instruction tuning are completely different things. Painting with watercolors does not make water paint. Nearly all popular local models are instruction tuned, but are not RLHF'd. The OAI GPT series are not the only LLMs in the world.

Man it really doesn't need to be said that RLHF is not the only way to instruct tune. The point of my comment was to say that was how GPT3.5 was instruct tuned, via RLHF through a question answer dataset. At least we have this needless nerd snipe so others won't be potentially misled by my careless quip.

It sounds like we both know that's the case, but there's a ton of incorrect info being shared in this thread re: RLHF and instruction tuning.

Sorry if it came off as more than looking to clarify it for folks coming across it.

Re: GitHub Copilot Chat Leaked Prompt

#184

Post by dang »

simonw wrote: post #103

Huh, this is the least interesting thing I've written about prompt injection in the last few weeks, but the only one to make it to the Hacker News homepage. Better recent posts: - Delimiters won’t save you from prompt injection - https://simonwillison.net/2023/May/11/delimiters-wont-save-y... - talks about why telling a model to follow delimiters like ``` won't protect against prompt injection, despite that being men…

Those posts are great! I've put https://news.ycombinator.com/item?id=35911595 ("Delimiters won’t save you from prompt injection") in the second-chance pool (https://news.ycombinator.com/pool, explained at https://news.ycombinator.com/item?id=26998308), so it will get a random placement on HN's front page. (I know you posted it earlier, but I prefer to spread the love by letting karma rain down on less-prolific submitters (love being one thing that isn't a power law).

I've emailed a repost invite to the submitter of https://news.ycombinator.com/item?id=35803564 ("Prompt Injection Explained"). Invited reposts go into the second-chance pool once they're submitted. If the article hasn't appeared after (say) a couple weeks or so, someone else is welcome to post it and email hn@ycombinator.com and we'll put it in the SCP.

I've emailed you a repost invite for https://news.ycombinator.com/item?id=35705159 ("The Dual LLM pattern for building AI assistants that can resist prompt injection"). It would be good to space these out, so maybe wait to use that link until a few days have gone by without one of your posts basking in front page glory?

Thanks for all the work figuring out this stuff and explaining it to the rest of us! It's amazing what a good writer can do when self-employed (https://news.ycombinator.com/item?id=35925266).

Re: GitHub Copilot Chat Leaked Prompt

#185

Post by cloudking »

rsync wrote: post #154
cloudking wrote: post #119

Earlier quoted context omitted.

The models are trained on text written by humans, so they respond and talk like humans.

Yes, but that is your parents' point: "And what kind of documents exist that begin with someone saying 'you are X, here are a bunch of rules for how X behaves', followed by a ..." Where, your parent asks, are all these reams of texts written in this manner ?

It's not that "you are X" type text has to be explicitly in the training data, it's that the model weights interpret "you are X" as an instruction that a human would receive as an emergent behavior after digesting a ton of human written text.

Re: GitHub Copilot Chat Leaked Prompt

#186

Post by AnthonyMouse »

JieJie wrote: post #169
skissane wrote: post #54

Earlier quoted context omitted.

> ChatGPT, Dall-e, etc all make assumptions about identity or politics but try to sidestep direct requests around those topics to appear more neutral... but the bias still exists in the model and affects the answers. In the case of ChatGPT, I’d love to know how much of the bias is in the original (pre)training data, and how much is due to OpenAI’s human trainers. It is so careful to avoid every bias which is condemne…

I wonder if it's as simple as adding a location check and updating the systemPrompt to tell the bot to assume the user is from _location_? How realistic is it if one says, "Hi, Bing. I'm from East London, please talk to me like you would to another East Londoner." Or is it obnoxiously stereotypical? I imagine, properly trained, that is something LLMs would excel at, communicating with people in their preferred mode o…

In many cases the difference between largely accurate and obnoxiously stereotypical is in the phrasing you use.

Re: GitHub Copilot Chat Leaked Prompt

#187

Post by stevenhuang »

alex_sf wrote: post #183
stevenhuang wrote: post #181

Earlier quoted context omitted.

Man it really doesn't need to be said that RLHF is not the only way to instruct tune. The point of my comment was to say that was how GPT3.5 was instruct tuned, via RLHF through a question answer dataset. At least we have this needless nerd snipe so others won't be potentially misled by my careless quip.

It sounds like we both know that's the case, but there's a ton of incorrect info being shared in this thread re: RLHF and instruction tuning. Sorry if it came off as more than looking to clarify it for folks coming across it.

Yes all that misinfo was what lead me to post a quick link. I could have been more clear anyways. Cheers.

Re: GitHub Copilot Chat Leaked Prompt

#189

Post by skissane »

emodendroket wrote: post #164
skissane wrote: post #151

Earlier quoted context omitted.

> I think the bigger issue is that the racial/sexist/etc content can be shocking and immediately put someone off using the product, which I doubt is the case for the output being “too American.” OpenAI didn't just fine-tune it to avoid blatant racial/sexist/etc content, they openly claim to have invested a lot of effort in fine-tuning it to avoid subtle biases in those areas. And to be honest, a lot of people do feel…

Maybe it’s different in international law but in US law “national origin” is just a term of art for race or ethnicity. Anyway, no matter how annoying seeing feet and inches may be for some users I don’t think it will have the same effect as the AI praising Hitler.

> Maybe it’s different in international law but in US law “national origin” is just a term of art for race or ethnicity.

Under Australian discrimination law, "national origin" means your country of birth, or the country of birth of your ancestors. [0] The Australian definition is (at least in part) derived from English discrimination law, as decided by the House of Lords in a 1972 case. [1] I believe the usual definition in international law is the same. "National origin" is related to "nationality/citizenship", but unlike the latter it is fixed at the time of birth – subsequent acts of naturalisation, renunciation or deprivation by change one's nationality, but they cannot change one's national origin.

I can't find a precise legal definition of the term under US law, but some US legal definitions appear largely identical to the Australian/English definition, see e.g. how District of Columbia law defines the term [2]

> Anyway, no matter how annoying seeing feet and inches may be for some users I don’t think it will have the same effect as the AI praising Hitler.

Well, as I said, OpenAI didn't just train ChatGPT to refuse to praise Hitler, they also went to a lot of effort to tune it to avoid subtle bias against certain identifiable groups. What I'm talking about isn't the same as the former but is clearly related to the latter

[0] Macabenta v Minister of State for Immigration and Multicultural Affairs [1998] FCA 1643; 90 FCR 202 https://jade.io/article/116118

[1] Ealing London Borough Council v Race Relations Board [1972] AC 342 https://www.casemine.com/judgement/uk/5a8ff8c960d03e7f57ecd6...

[2] https://ohr.dc.gov/sites/default/files/dc/sites/ohr/publicat...

Re: GitHub Copilot Chat Leaked Prompt

#190

Post by slg »

naet wrote: post #19

I think that a lot of the limits placed on these models / chat services don't do much to remove underlying bias but rather attempt to obfuscate them from the general public. ChatGPT, Dall-e, etc all make assumptions about identity or politics but try to sidestep direct requests around those topics to appear more neutral... but the bias still exists in the model and affects the answers.

>ChatGPT, Dall-e, etc all make assumptions about identity or politics but try to sidestep direct requests around those topics to appear more neutral... but the bias still exists in the model and affects the answers.

Honestly, that might be the most human trait they have.

Post reply on HN