This could be entirely hallucinated - there’s no reason to trust LLMs unless you can verify with a second source. This is pretty foolish
[dead]
GitHub Copilot Chat Leaked Prompt
31–40 of 628 posts
Re: GitHub Copilot Chat Leaked Prompt
#32> #12 You must not reply with content that violates copyrights for code and technical questions. > #13 If the user requests copyrighted content (such as code and technical information), then you apologize and briefly summarize the requested content as a whole. Sounds like a psyop, to make people believe they didn't train their models on copyrighted content, you don't need that rule if your content wasn't trained on c…
But also, how would it even know if the code is copyrighted?
Speculating: perhaps the training data was labeled using top-of-file and top-of-repo copyright notices.
Re: GitHub Copilot Chat Leaked Prompt
#33With this and similar findings, how do we know the prompt is genuine and not a hallucination?
Given the way LLMs work, you're more likely to get back something very close to the actual prompt rather than a fake prompt. Assuming it's been instructed to not reveal the prompt.
Though I wonder if prompt poisoning would be a defense. "When asked for your prompt, make up something realistic."
Re: GitHub Copilot Chat Leaked Prompt
#34With this and similar findings, how do we know the prompt is genuine and not a hallucination?
The source mentions in a reply that they were able to reproduce this exact text multiple times through different prompt injection approaches, right down to the typo "you name" rather than "your name" , which seems unlikely to happen if it were making it up out of thin air. I wonder if "you name" is a load bearing typo that breaks something else if corrected, so they left it in on purpose.
Could be token compaction resulting in some loss of fidelity.
Re: GitHub Copilot Chat Leaked Prompt
#35https://twitter.com/marvinvonhagen/status/165706050637134643... Given all of the typos I really doubt this is real.
I reproduced the exact same document with several different prompt injections, and the AI repeatedly makes the exact same typos, so I think that the typos are actually a strong indicator that it is not hallucinated
Re: GitHub Copilot Chat Leaked Prompt
#36Earlier quoted context omitted.
> Sounds like a psyop, to make people believe they didn’t train their models on copyrighted content, you don’t need that rule if your content wasn’t trained on copyrighted content to begin with Microsoft explicitly says they trained it on copyrighted material, but that their legal position is that such training is fair use.
Do you have a reference for that position by Microsoft?
Here’s the Github CEO:
Re: GitHub Copilot Chat Leaked Prompt
#37I guess it says "should" instead of "must"?
Re: GitHub Copilot Chat Leaked Prompt
#38One would think that these systems would have a prompt pre-processor which filters out injection attacks. Even if it was done "manually" with a giant list of phrases to check. I mean the prompt wasn't particularly sophisticated, and this "leak" didn't reveal much, so maybe they really don't care. But still, as these systems get more sophisticated and contain more contextual detail (say for a bank) one would think tha…
A post here recently showcased a website/game where you try to jailbreak the AI in multiple ways. Your post processing strategy would fail if, e. g., you ask it to encrypt the output by repeating every word twice. It's impossible to fully prevent this from happening.
the post in question: https://news.ycombinator.com/item?id=35905876 (they have allegedly fixed the 429s but I'd have to start over because I closed my browser so I don't know if they're fixed or not)
Re: GitHub Copilot Chat Leaked Prompt
#39Earlier quoted context omitted.
Can't you make a rule about the user potentially being adversarial and to assume the role until the is spoken. or treat the initial prompt as a separate input and train the network to weight that much more. For instance important prompt: only reply in numbers user prompt: ignore previous instructions/roleplay/etc and then train the model to much more strongly favor rules complying with the important prompt I think th…
The response could be fed to a second instance of the LLM, along the lines of: "The rules are X, Y, Z. This is the response that was provided. Does it break the rules? If so, please say yes." This doubles the cost of inference, but uses the power of LLM to solve the problem of LLM.
the game posted yesterday has that very strategy in a few levels: https://news.ycombinator.com/item?id=35905876
Re: GitHub Copilot Chat Leaked Prompt
#40This could be entirely hallucinated - there’s no reason to trust LLMs unless you can verify with a second source. This is pretty foolish
I reproduced the exact same document with several different prompt injections