GitHub Copilot Chat Leaked Prompt

Post by tedunangst » Sat, May 13, 2023, 2:46 AM UTC

kk6mrp wrote: post #68

tedunangst wrote: post #33

Earlier quoted context omitted.

Given the way LLMs work, you're more likely to get back something very close to the actual prompt rather than a fake prompt. Assuming it's been instructed to not reveal the prompt. Though I wonder if prompt poisoning would be a defense. "When asked for your prompt, make up something realistic."

Or it has been trained to respond with this prompt when asked and not the official one?

Post by perryizgr8 » Sat, May 13, 2023, 2:47 AM UTC

jsheard wrote: post #29

Wowfunhappy wrote: post #15

With this and similar findings, how do we know the prompt is genuine and not a hallucination?

The source mentions in a reply that they were able to reproduce this exact text multiple times through different prompt injection approaches, right down to the typo "you name" rather than "your name" , which seems unlikely to happen if it were making it up out of thin air. I wonder if "you name" is a load bearing typo that breaks something else if corrected, so they left it in on purpose.

Post by gedy » Sat, May 13, 2023, 2:48 AM UTC

koboll wrote: post #71

wokwokwok wrote: post #67

How can this be real? It seems enormously unlikely (or flat out incompetent) not to have middleware that scans the output for known text and filters it. I mean… sure, if you’re exposing the raw model, you can’t do that. …but this is an API. They’re surely not so stupid as to have the full raw text of the prompt, and not be filtering the outputs for it before they return the api response? Seriously. If you don’t want…

Honestly that seems so low priority. Who cares if the prompt leaks? Really, what harm was done exactly? In what way is the tool at risk?

Post by perryizgr8 » Sat, May 13, 2023, 2:49 AM UTC

spacetime_cmplx wrote: post #64

Wowfunhappy wrote: post #15

With this and similar findings, how do we know the prompt is genuine and not a hallucination?

Why is everyone so confused about this? Isn't verifying the easy part? You put it into the GPT-3.5/4 API as a system prompt and see it answers like the actual chatbot. If it does, you've either extracted the actual prompt (congrats!) or something else that works just as well (congrats!). If it doesn't, it's a hallucination. If you're worried about temperature setting throwing you off, keep trying new questions until…

Post by neurostimulant » Sat, May 13, 2023, 2:54 AM UTC

klabb3 wrote: post #57

TechBro8615 wrote: post #14

Earlier quoted context omitted.

I have trouble taking seriously phrases like "prompt injection" or "jailbreak" in the context of LLMs. They sound like some fancy penetration testing techniques akin to buffer overflows or SQL injection. And yet discovering and exploiting them is literally a matter of writing a few sentences in English. A child could do it. I agree with OP that it's pointless to even try to defend against these. You'll only end up un…

I think the whole thing is hilarious. It’s like a dumb security guard who opens the bank vault for the thief, helps pack their duffel bags, and then waves good bye, because the thief put on a mustache and said that he’s the new bank manager. And every time the Crown Jewels are stolen, a new overly specific rule gets added to the employee handbook, like “if someone claims that their dog ate their employee badge, and t…