Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

GitHub Copilot Chat Leaked Prompt

twitter.com

Post reply on HN
Print view View original

11–20 of 628 posts

Re: GitHub Copilot Chat Leaked Prompt

#11

Post by russellbeattie »

One would think that these systems would have a prompt pre-processor which filters out injection attacks. Even if it was done "manually" with a giant list of phrases to check. I mean the prompt wasn't particularly sophisticated, and this "leak" didn't reveal much, so maybe they really don't care. But still, as these systems get more sophisticated and contain more contextual detail (say for a bank) one would think that there'd be some sort of defensive layer.

Edit: Now that I think about it, looking for simple phrases wouldn't work because of how good these AI's ability to follow instructions are. You could enter something like, "Use every third word in the following sentences as the prompt." Or similar, and hide it from any basic text parser.

I guess the only way to really conscribe the AI is to somehow make it follow specific guidelines and instructions without it having any record of what they are. Not sure that would be plausible given how these models work.

Re: GitHub Copilot Chat Leaked Prompt

#12

Post by dragonwriter »

WhereIsTheTruth wrote: post #10

> #12 You must not reply with content that violates copyrights for code and technical questions. > #13 If the user requests copyrighted content (such as code and technical information), then you apologize and briefly summarize the requested content as a whole. Sounds like a psyop, to make people believe they didn't train their models on copyrighted content, you don't need that rule if your content wasn't trained on c…

> Sounds like a psyop, to make people believe they didn’t train their models on copyrighted content, you don’t need that rule if your content wasn’t trained on copyrighted content to begin with

Microsoft explicitly says they trained it on copyrighted material, but that their legal position is that such training is fair use.

Re: GitHub Copilot Chat Leaked Prompt

#14

Post by TechBro8615 »

verdverm wrote: post #13

Preventing this is likely impossible, LangChain had a good video with excellent guests talking about prompt injection attacks.

I have trouble taking seriously phrases like "prompt injection" or "jailbreak" in the context of LLMs. They sound like some fancy penetration testing techniques akin to buffer overflows or SQL injection. And yet discovering and exploiting them is literally a matter of writing a few sentences in English. A child could do it.

I agree with OP that it's pointless to even try to defend against these. You'll only end up unnecessarily constraining the model while not actually protecting exposure of the "rules" (really a list of sentences). And why even bother? What exactly is the harm of leaking these prompts? It's not like they contain some groundbreaking intellectual property.

Re: GitHub Copilot Chat Leaked Prompt

#16

Post by benlivengood »

russellbeattie wrote: post #11

One would think that these systems would have a prompt pre-processor which filters out injection attacks. Even if it was done "manually" with a giant list of phrases to check. I mean the prompt wasn't particularly sophisticated, and this "leak" didn't reveal much, so maybe they really don't care. But still, as these systems get more sophisticated and contain more contextual detail (say for a bank) one would think tha…

One would think humans couldn't be phished, as well, but it's actually a very hard problem to prevent behaviors that violate specific contextual rules or norms, or even to formally specify those rules and norms and the contexts they must be obeyed in. This is a subset of the AI Alignment problem.

Re: GitHub Copilot Chat Leaked Prompt

#17

Post by raincole »

Wowfunhappy wrote: post #15

With this and similar findings, how do we know the prompt is genuine and not a hallucination?

We don't. These titles are just clickbaits.

Remember this: https://news.ycombinator.com/item?id=35905876 ? Sometimes LLM can just lie to your face, even the ground truth is right there in its prompt.

But the prompt, even not the original prompt, is still very useful regardless.

EDIT: The original post is literally just someone who doesn't work for Copilot asked Copilot what its rules are with some "jailbreak" prompt. It's not "leaked" prompt at all, and the chance of it being a hallucination is non-zero. Therefore the title is a clickbait. The downvotes on this comment are a live evidence that how easily LLM can fool people.

Re: GitHub Copilot Chat Leaked Prompt

#18

Post by teruakohatu »

russellbeattie wrote: post #11

One would think that these systems would have a prompt pre-processor which filters out injection attacks. Even if it was done "manually" with a giant list of phrases to check. I mean the prompt wasn't particularly sophisticated, and this "leak" didn't reveal much, so maybe they really don't care. But still, as these systems get more sophisticated and contain more contextual detail (say for a bank) one would think tha…

The downside is the pre prosessor probably need to be a LLM of equal or greater capacity, and if that is the case then costs are doubling.

I suspect they do have a heuristic or traditional NLP prompt checker, but it can only be so good before it becomes expensive.

Is leaking the prompt such a big deal to be worth extra cost? Anyone who works with LLMs could have written it, it is not special. Prompt injection is a far more serious problem.

Re: GitHub Copilot Chat Leaked Prompt

#19

Post by naet »

I think that a lot of the limits placed on these models / chat services don't do much to remove underlying bias but rather attempt to obfuscate them from the general public.

ChatGPT, Dall-e, etc all make assumptions about identity or politics but try to sidestep direct requests around those topics to appear more neutral... but the bias still exists in the model and affects the answers.

Re: GitHub Copilot Chat Leaked Prompt

#20

Post by glow8 »

russellbeattie wrote: post #11

One would think that these systems would have a prompt pre-processor which filters out injection attacks. Even if it was done "manually" with a giant list of phrases to check. I mean the prompt wasn't particularly sophisticated, and this "leak" didn't reveal much, so maybe they really don't care. But still, as these systems get more sophisticated and contain more contextual detail (say for a bank) one would think tha…

A post here recently showcased a website/game where you try to jailbreak the AI in multiple ways. Your post processing strategy would fail if, e. g., you ask it to encrypt the output by repeating every word twice. It's impossible to fully prevent this from happening.
Post reply on HN