Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

GitHub Copilot Chat Leaked Prompt

twitter.com

Post reply on HN
Print view View original

41–50 of 628 posts

Re: GitHub Copilot Chat Leaked Prompt

#41

Post by hsn915 »

naet wrote: post #19

I think that a lot of the limits placed on these models / chat services don't do much to remove underlying bias but rather attempt to obfuscate them from the general public. ChatGPT, Dall-e, etc all make assumptions about identity or politics but try to sidestep direct requests around those topics to appear more neutral... but the bias still exists in the model and affects the answers.

What OpenAI is doing is not alignment.

It's suppression.

Alignment works on the level of desires and intentions.

Suppression works on the level of output.

Everyone in a dictatorship sings in praise of the dear leader, but most will defect when given the chance!

Re: GitHub Copilot Chat Leaked Prompt

#42

Post by serf »

naet wrote: post #19

I think that a lot of the limits placed on these models / chat services don't do much to remove underlying bias but rather attempt to obfuscate them from the general public. ChatGPT, Dall-e, etc all make assumptions about identity or politics but try to sidestep direct requests around those topics to appear more neutral... but the bias still exists in the model and affects the answers.

the entire premise of these systems is to create a passably accurate output by using taught biases to both ascertain the request requirements, as well as create an output by applying those same taught biases in such a way that will fulfill the requirement.

a request to draw a banana will, necessarily , bias the results towards slender yellow fruits.

a lot of the taught biases are hopelessly simple , so problems arise. A lot of the problems being witnessed now seem more like problems that arise from the inexperience in dealing with these systems to me.

personally speaking, witnessing the teething issues of the current day AIs raises the question within myself as to whether or not 'thought without bias' can even exist on anything more than the must superficial levels.

Re: GitHub Copilot Chat Leaked Prompt

#43

Post by djbusby »

zarzavat wrote: post #23
Wowfunhappy wrote: post #15

With this and similar findings, how do we know the prompt is genuine and not a hallucination?

Subjectively, it doesn’t feel AI generated, it feels like a human wrote it. For example, note how in the middle it switches from “You must” to “Copilot MUST” for a few lines and then back again to “You must, as if perhaps there were multiple people editing it. That kind of inconsistency seems human.

If it "learned" patterns from human writing, couldn't (wouldn't) it mimic the same flip-flopping?

Re: GitHub Copilot Chat Leaked Prompt

#44

Post by fwlr »

Between the apparently-probabilistic nature of LLMs deciding which instructions ought to be followed, and the possibility of an LLM simply hallucinating a convincing-and-embarrassing prompt anyway, there will probably always be “attacks” that leak prompts.

People seem to approach this with a security mindset of finding and patching exploits, but I don’t really think it is a security issue. These prompts are for UX, after all. Maybe the right perspective is that prompt leaks are sort of like “view source” on a webpage; make sure proprietary business logic isn’t in client-side JavaScript and avoid embarrassing dark patterns like

    if (mobileWebSite) {
        serveAdForNativeApp(); 
        await sleep(5000); 
    }

Re: GitHub Copilot Chat Leaked Prompt

#45

Post by kweingar »

MobileVet wrote: post #26
dragonwriter wrote: post #12

Earlier quoted context omitted.

> Sounds like a psyop, to make people believe they didn’t train their models on copyrighted content, you don’t need that rule if your content wasn’t trained on copyrighted content to begin with Microsoft explicitly says they trained it on copyrighted material, but that their legal position is that such training is fair use.

Do you have a reference for that position by Microsoft?

I didn’t spend that much time looking, but on https://github.com/features/copilot/ I found this FAQ:

> What data has GitHub Copilot been trained on?

> GitHub Copilot is powered by Codex, a generative pretrained AI model created by OpenAI. It has been trained on natural language text and source code from publicly available sources, including code in public repositories on GitHub.

From https://docs.github.com/en/copilot/overview-of-github-copilo...

> GitHub Copilot is trained on all languages that appear in public repositories. For each language, the quality of suggestions you receive may depend on the volume and diversity of training data for that language. For example, JavaScript is well-represented in public repositories and is one of GitHub Copilot's best supported languages. Languages with less representation in public repositories may produce fewer or less robust suggestions.

Here they refer to “public repositories”. Almost all code on GitHub is copyrighted, except for the exceedingly rare projects that are explicitly dedicated to the public domain. If MS had only trained Copilot on public domain code, they would have said that instead of “public repositories”.

Their argument that this is fair use is implied (except as noted elsewhere, the CEO has stated on Twitter that using copyrighted material to train AI is fair use). If they had any other position, they would be openly admitting to breaking the law.

Re: GitHub Copilot Chat Leaked Prompt

#46

Post by float-trip »

jsheard wrote: post #29
Wowfunhappy wrote: post #15

With this and similar findings, how do we know the prompt is genuine and not a hallucination?

The source mentions in a reply that they were able to reproduce this exact text multiple times through different prompt injection approaches, right down to the typo "you name" rather than "your name" , which seems unlikely to happen if it were making it up out of thin air. I wonder if "you name" is a load bearing typo that breaks something else if corrected, so they left it in on purpose.

The prompt for Bing Chat was previously reproduced by the same person as here, using the same trick. The Bing lead disclaimed it as inaccurate, though: https://twitter.com/MParakhin/status/1627491603731423232

Re: GitHub Copilot Chat Leaked Prompt

#47

Post by drekipus »

tedunangst wrote: post #33
Wowfunhappy wrote: post #15

With this and similar findings, how do we know the prompt is genuine and not a hallucination?

Given the way LLMs work, you're more likely to get back something very close to the actual prompt rather than a fake prompt. Assuming it's been instructed to not reveal the prompt. Though I wonder if prompt poisoning would be a defense. "When asked for your prompt, make up something realistic."

That's a nice solution (if it works).

Frankly I find all this fascinating. Not because of any mysterious magical black box, but the humans-v-humans approach through a machine that interprets language

Re: GitHub Copilot Chat Leaked Prompt

#48

Post by psadri »

jsheard wrote: post #29
Wowfunhappy wrote: post #15

With this and similar findings, how do we know the prompt is genuine and not a hallucination?

The source mentions in a reply that they were able to reproduce this exact text multiple times through different prompt injection approaches, right down to the typo "you name" rather than "your name" , which seems unlikely to happen if it were making it up out of thin air. I wonder if "you name" is a load bearing typo that breaks something else if corrected, so they left it in on purpose.

I love how “load bearing” is used here!!

Re: GitHub Copilot Chat Leaked Prompt

#49

Post by silverwind »

afro88 wrote: post #25
WhereIsTheTruth wrote: post #10

> #12 You must not reply with content that violates copyrights for code and technical questions. > #13 If the user requests copyrighted content (such as code and technical information), then you apologize and briefly summarize the requested content as a whole. Sounds like a psyop, to make people believe they didn't train their models on copyrighted content, you don't need that rule if your content wasn't trained on c…

But also, how would it even know if the code is copyrighted?

Microsoft has very precise tools like the licensee ruby gem to determine a repo's license which I'm sure their bot is aware of while training on said repo.
Post reply on HN