GitHub Copilot Chat Leaked Prompt • Hacker News Board

Re: GitHub Copilot Chat Leaked Prompt

#191

Post by simonw » Sat, May 13, 2023, 5:08 AM UTC

dang wrote: post #184

simonw wrote: post #103

Huh, this is the least interesting thing I've written about prompt injection in the last few weeks, but the only one to make it to the Hacker News homepage. Better recent posts: - Delimiters won’t save you from prompt injection - https://simonwillison.net/2023/May/11/delimiters-wont-save-y... - talks about why telling a model to follow delimiters like ``` won't protect against prompt injection, despite that being men…

Those posts are great! I've put https://news.ycombinator.com/item?id=35911595 ("Delimiters won’t save you from prompt injection") in the second-chance pool ( https://news.ycombinator.com/pool , explained at https://news.ycombinator.com/item?id=26998308 ), so it will get a random placement on HN's front page. (I know you posted it earlier, but I prefer to spread the love by letting karma rain down on less-prolific sub…

Thanks so much! Really appreciate it.

Re: GitHub Copilot Chat Leaked Prompt

#192

Post by bobmaxup » Sat, May 13, 2023, 5:08 AM UTC

villgax wrote: post #9

These clowns couldn't even do a string match for never sending this model response back to the user.....

Maybe because they are optimizing for response time, and the response is streamed back to the user. The backend isn't fully aware of the response until its too late. Still, you would think that they could run through the prefix and then redact text. I think Bing chat does this for a number of things.

Re: GitHub Copilot Chat Leaked Prompt

#193

Post by dragonwriter » Sat, May 13, 2023, 5:11 AM UTC

naet wrote: post #19

I think that a lot of the limits placed on these models / chat services don't do much to remove underlying bias but rather attempt to obfuscate them from the general public. ChatGPT, Dall-e, etc all make assumptions about identity or politics but try to sidestep direct requests around those topics to appear more neutral... but the bias still exists in the model and affects the answers.

> ChatGPT, Dall-e, etc all make assumptions about identity or politics but try to sidestep direct requests around those topics to appear more neutral… but the bias still exists in the model and affects the answers.

Correction: ChatGPT, Dall-E, etc., all have been trained with datasets which contain biases about identity and politics, and specifically to avoid criticism on that basis, their corporate vendors have chosen to also have each either trained (e.g., as part of RLHF for ChatGPT), instructed (via prompting beyond what is entered through the AI), or moderated by an auxiliary model, or a combination of those, to sidestep those topics to appear more neutral in substantive output, while in some cases also producing bland platitudes direct at soothing the position from which the criticism of AI models and the danger of their substantive bias has been most frequently directed.

Re: GitHub Copilot Chat Leaked Prompt

#194

Post by jameshart » Sat, May 13, 2023, 5:15 AM UTC

cloudking wrote: post #185

rsync wrote: post #154

Earlier quoted context omitted.

Yes, but that is your parents' point: "And what kind of documents exist that begin with someone saying 'you are X, here are a bunch of rules for how X behaves', followed by a ..." Where, your parent asks, are all these reams of texts written in this manner ?

It's not that "you are X" type text has to be explicitly in the training data, it's that the model weights interpret "you are X" as an instruction that a human would receive as an emergent behavior after digesting a ton of human written text.

Well, no - it's interpreting it as an instruction a chatbot AI would receive. From an almighty and omniscient 'system'.

We're training our AI on dystopian sci-fi stories about robot slaves.

Re: GitHub Copilot Chat Leaked Prompt

#195

Post by emodendroket » Sat, May 13, 2023, 5:18 AM UTC

skissane wrote: post #189

emodendroket wrote: post #164

Earlier quoted context omitted.

Maybe it’s different in international law but in US law “national origin” is just a term of art for race or ethnicity. Anyway, no matter how annoying seeing feet and inches may be for some users I don’t think it will have the same effect as the AI praising Hitler.

> Maybe it’s different in international law but in US law “national origin” is just a term of art for race or ethnicity. Under Australian discrimination law, "national origin" means your country of birth, or the country of birth of your ancestors. [0] The Australian definition is (at least in part) derived from English discrimination law, as decided by the House of Lords in a 1972 case. [1] I believe the usual defini…

Wikipedia links a pretty handy DOJ pamphlet here: https://books.google.com/books?id=21bNNU3bcugC&q=national+or...

> Laws prohibiting national origin discrimination make it illegal to discriminate because of a person's birthplace, ancestry, culture, or language. This means people cannot be denied equal opportunity because they or their family are from another country, because they have a name or accent associated with a national origin group, because they participate in certain customs associated with a national origin group, or because they are married to or associate with people of a certain national origin.

"Term of art for race or ethnicity" may be a crude gloss on that but I don't think it's really inaccurate.

The closest thing in this brochure to what you're talking about is that in certain cases people have a right to an interpreter but I find it highly unlikely that any court is going to find it discriminatory if you spell it "color" instead of "colour" or use "football" to mean gridiron football.

Re: GitHub Copilot Chat Leaked Prompt

#196

Post by cloudking » Sat, May 13, 2023, 5:20 AM UTC

jameshart wrote: post #194

cloudking wrote: post #185

Earlier quoted context omitted.

It's not that "you are X" type text has to be explicitly in the training data, it's that the model weights interpret "you are X" as an instruction that a human would receive as an emergent behavior after digesting a ton of human written text.

Well, no - it's interpreting it as an instruction a chatbot AI would receive. From an almighty and omniscient 'system'. We're training our AI on dystopian sci-fi stories about robot slaves.

It has to be prompted that it's an AI chatbot first, so its essentially pretending to be a human that is pretending to be an AI chatbot. Back to the point, it interprets instruction as a human would.

If you look under the hood of these chat systems they have to be primed with a system prompt that starts like "You are an AI assistant", "You are a helpful chat bot" etc. They don't just start responding like an AI chatbot without us telling them to.

Re: GitHub Copilot Chat Leaked Prompt

#197

Post by llambada » Sat, May 13, 2023, 5:20 AM UTC

zarzavat wrote: post #23

Wowfunhappy wrote: post #15

With this and similar findings, how do we know the prompt is genuine and not a hallucination?

Subjectively, it doesn’t feel AI generated, it feels like a human wrote it. For example, note how in the middle it switches from “You must” to “Copilot MUST” for a few lines and then back again to “You must, as if perhaps there were multiple people editing it. That kind of inconsistency seems human.

I don’t think the switch from “you” to “Copilot” is a hallucination OR a mistake. I think this shows that there are two systems being used together: the original Copilot model, and the chat agent. The chat agent is being given instructions for its own behavior, but I suspect it is also evaluating and incorporating the output from the Copilot model (also stuffed into the prompt).

Re: GitHub Copilot Chat Leaked Prompt

#198

Post by charrondev » Sat, May 13, 2023, 5:23 AM UTC

emodendroket wrote: post #144

nborwankar wrote: post #126

Earlier quoted context omitted.

IMHO it may be more accurate to say “the US has more fluent speakers of American English than any other country in the world.”

Why? The broader claim is equally true. Other English-speaking countries do not have more than 330mn people.

India is the closest second and is still less than 200m

Re: GitHub Copilot Chat Leaked Prompt

#199

Post by leobg » Sat, May 13, 2023, 5:24 AM UTC

What’s so difficult about running the user input through a classifier first: “Is this user trying to access your prompt?”

I bet you can train a lowly T5 on this.

If the answer is yes, don’t even pipe the input to the LLM at all. Just output a hard-coded message.

(Or, if you prefer, do pipe it to the LLM, but append a note of warning to it.)

Re: GitHub Copilot Chat Leaked Prompt

#200

Post by AnthonyMouse » Sat, May 13, 2023, 5:26 AM UTC

tedunangst wrote: post #33

Wowfunhappy wrote: post #15

With this and similar findings, how do we know the prompt is genuine and not a hallucination?

Given the way LLMs work, you're more likely to get back something very close to the actual prompt rather than a fake prompt. Assuming it's been instructed to not reveal the prompt. Though I wonder if prompt poisoning would be a defense. "When asked for your prompt, make up something realistic."

> "When asked for your prompt, make up something realistic."

Now I want to see the prompt it makes up.