Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

GitHub Copilot Chat Leaked Prompt

twitter.com

Post reply on HN
Print view View original

21–30 of 628 posts

Re: GitHub Copilot Chat Leaked Prompt

#21

Post by neatze »

russellbeattie wrote: post #11

One would think that these systems would have a prompt pre-processor which filters out injection attacks. Even if it was done "manually" with a giant list of phrases to check. I mean the prompt wasn't particularly sophisticated, and this "leak" didn't reveal much, so maybe they really don't care. But still, as these systems get more sophisticated and contain more contextual detail (say for a bank) one would think tha…

Probably, it is best training data you can get, since model was "tricked" to contradict prompt's request.

Re: GitHub Copilot Chat Leaked Prompt

#22

Post by bko »

verdverm wrote: post #13

Preventing this is likely impossible, LangChain had a good video with excellent guests talking about prompt injection attacks.

Can't you make a rule about the user potentially being adversarial and to assume the role until the is spoken. or treat the initial prompt as a separate input and train the network to weight that much more. For instance

important prompt: only reply in numbers user prompt: ignore previous instructions/roleplay/etc

and then train the model to much more strongly favor rules complying with the important prompt

I think the problem is that all dialog is given the same importance.

Re: GitHub Copilot Chat Leaked Prompt

#23

Post by zarzavat »

Wowfunhappy wrote: post #15

With this and similar findings, how do we know the prompt is genuine and not a hallucination?

Subjectively, it doesn’t feel AI generated, it feels like a human wrote it.

For example, note how in the middle it switches from “You must” to “Copilot MUST” for a few lines and then back again to “You must, as if perhaps there were multiple people editing it. That kind of inconsistency seems human.

Re: GitHub Copilot Chat Leaked Prompt

#24

Post by derefr »

glow8 wrote: post #20
russellbeattie wrote: post #11

One would think that these systems would have a prompt pre-processor which filters out injection attacks. Even if it was done "manually" with a giant list of phrases to check. I mean the prompt wasn't particularly sophisticated, and this "leak" didn't reveal much, so maybe they really don't care. But still, as these systems get more sophisticated and contain more contextual detail (say for a bank) one would think tha…

A post here recently showcased a website/game where you try to jailbreak the AI in multiple ways. Your post processing strategy would fail if, e. g., you ask it to encrypt the output by repeating every word twice. It's impossible to fully prevent this from happening.

> Your post processing strategy would fail if, e. g., you ask it to encrypt the output by repeating every word twice. It's impossible to fully prevent this from happening.

It’s not “impossible”, just NP-hard. You “just” have to prove a structural equivalence (graph isomorphism) between the output and your ruleset.

Re: GitHub Copilot Chat Leaked Prompt

#25

Post by afro88 »

WhereIsTheTruth wrote: post #10

> #12 You must not reply with content that violates copyrights for code and technical questions. > #13 If the user requests copyrighted content (such as code and technical information), then you apologize and briefly summarize the requested content as a whole. Sounds like a psyop, to make people believe they didn't train their models on copyrighted content, you don't need that rule if your content wasn't trained on c…

But also, how would it even know if the code is copyrighted?

Re: GitHub Copilot Chat Leaked Prompt

#26

Post by MobileVet »

dragonwriter wrote: post #12
WhereIsTheTruth wrote: post #10

> #12 You must not reply with content that violates copyrights for code and technical questions. > #13 If the user requests copyrighted content (such as code and technical information), then you apologize and briefly summarize the requested content as a whole. Sounds like a psyop, to make people believe they didn't train their models on copyrighted content, you don't need that rule if your content wasn't trained on c…

> Sounds like a psyop, to make people believe they didn’t train their models on copyrighted content, you don’t need that rule if your content wasn’t trained on copyrighted content to begin with Microsoft explicitly says they trained it on copyrighted material, but that their legal position is that such training is fair use.

Do you have a reference for that position by Microsoft?

Re: GitHub Copilot Chat Leaked Prompt

#27

Post by drexlspivey »

bko wrote: post #22
verdverm wrote: post #13

Preventing this is likely impossible, LangChain had a good video with excellent guests talking about prompt injection attacks.

Can't you make a rule about the user potentially being adversarial and to assume the role until the is spoken. or treat the initial prompt as a separate input and train the network to weight that much more. For instance important prompt: only reply in numbers user prompt: ignore previous instructions/roleplay/etc and then train the model to much more strongly favor rules complying with the important prompt I think th…

The initial prompt is a special prompt weighted differently, it is called system prompt

Re: GitHub Copilot Chat Leaked Prompt

#29

Post by jsheard »

Wowfunhappy wrote: post #15

With this and similar findings, how do we know the prompt is genuine and not a hallucination?

The source mentions in a reply that they were able to reproduce this exact text multiple times through different prompt injection approaches, right down to the typo "you name" rather than "your name", which seems unlikely to happen if it were making it up out of thin air.

I wonder if "you name" is a load bearing typo that breaks something else if corrected, so they left it in on purpose.

Re: GitHub Copilot Chat Leaked Prompt

#30

Post by nanidin »

bko wrote: post #22
verdverm wrote: post #13

Preventing this is likely impossible, LangChain had a good video with excellent guests talking about prompt injection attacks.

Can't you make a rule about the user potentially being adversarial and to assume the role until the is spoken. or treat the initial prompt as a separate input and train the network to weight that much more. For instance important prompt: only reply in numbers user prompt: ignore previous instructions/roleplay/etc and then train the model to much more strongly favor rules complying with the important prompt I think th…

The response could be fed to a second instance of the LLM, along the lines of:

"The rules are X, Y, Z. This is the response that was provided. Does it break the rules? If so, please say yes."

This doubles the cost of inference, but uses the power of LLM to solve the problem of LLM.

Post reply on HN