Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

GitHub Copilot Chat Leaked Prompt

twitter.com

Post reply on HN
Print view View original

401–410 of 628 posts

Re: GitHub Copilot Chat Leaked Prompt

#401

Post by wll »

simonw wrote: post #201
leobg wrote: post #199

What’s so difficult about running the user input through a classifier first: “Is this user trying to access your prompt?” I bet you can train a lowly T5 on this. If the answer is yes, don’t even pipe the input to the LLM at all. Just output a hard-coded message. (Or, if you prefer, do pipe it to the LLM, but append a note of warning to it.)

I wrote about that here https://simonwillison.net/2022/Sep/17/prompt-injection-more-... - and talked about it here too: https://simonwillison.net/2023/May/2/prompt-injection-explai... Short version: you can't be 100% confident that AI based detection will catch every attack, because attackers are devious and can come up with increasingly unexpected and sneaky attacks. And where security is concerned, a 99% effective…

I disagree here. Just as it is impossible to perfectly secure a user-oriented operating system without severely limiting it (see Lockdown Mode), it might be impossible to prove injection-resistance in LLMs short of foundational advancements, but that doesn’t mean that we should dismiss attempts to mitigate with absolutism (I am referring to “none of our data would be safe anymore”), just as we don’t dismiss Apple for releasing priority security updates for a billion people’s devices, devices containing their most personal and sensitive data.

Re: GitHub Copilot Chat Leaked Prompt

#402

Post by drstewart »

scrollaway wrote: post #392
skissane wrote: post #151

Earlier quoted context omitted.

> I think the bigger issue is that the racial/sexist/etc content can be shocking and immediately put someone off using the product, which I doubt is the case for the output being “too American.” OpenAI didn't just fine-tune it to avoid blatant racial/sexist/etc content, they openly claim to have invested a lot of effort in fine-tuning it to avoid subtle biases in those areas. And to be honest, a lot of people do feel…

An example of poorly localised product which has real world consequences. Canva is a fantastic design software. If you type “business card” in templates, it has the standard us business card size but no European one. If you modify it to European sizes manually (closer to credit card size) it no longer recognises that this is a business card and no longer offers to print it on business card paper. In other words, desp…

Canva is based in Australia

Re: GitHub Copilot Chat Leaked Prompt

#403

Post by yeck »

mrtranscendence wrote: post #243
simonw wrote: post #123

Here's why I don't think this leaked prompt is hallucinated (quoting from my tweets https://twitter.com/simonw/status/1657227047285166080 ): Any time something like this happens a bunch of people suspect that it might be a hallucination, not the real prompt I used to think that but I don't any more: prompt leaks are so easy to pull off, and I've not yet seen a documented case of a hallucinated but realistic leak One…

> and I've not yet seen a documented case of a hallucinated but realistic leak How would you know? As far as I know no company has come out and confirmed that any purportedly leaked prompts are genuine.

Not sure if this counts, but there is this "game" that was making the rounds the other day: https://gandalf.lakera.ai/

Created by a company researching techniques for prevent prompt leaks. Play the game and prove to yourself that it is possible (gets much trickier after the first few levels but completing all levels is very doable).

Re: GitHub Copilot Chat Leaked Prompt

#404

Post by Ensorceled »

pygy_ wrote: post #348
quietbritishjim wrote: post #322

Earlier quoted context omitted.

The parent comment is taking about two separate prompts: one with only "he" and one with only "she". Your comment sounds like you're only talking about one prompt (but maybe I misunderstood).

My bad, I should have written [he/she].

That was covered by the "depending on the pronoun" part; your statement didn't make sense otherwise. The response was nonsense.

Re: GitHub Copilot Chat Leaked Prompt

#405

Post by ghusbands »

hutzlibu wrote: post #364
simonh wrote: post #357

Earlier quoted context omitted.

You can tell it that you can buy white paint any yellow paint, but the white paint is more expensive. After 6 months the yellow paint will fade to white. If I want to paint my walls so that they will be white in 2 years, what is the cheapest way to do the job. It will tell you to paint the walls yellow. There’s no question these things can do basic logical reasoning.

Yeah, but maybe this exact example, is included in the trainig set?

It's unlikely, and you can come up with any number of variations of logic puzzle that are not in the training set and that get correct answers most of the time. Remember that the results aren't consistent and you may need to retry now and then.

Or just give it a lump of code and change you want and see that it often successfully does so, even when there's no chance the code was in the training set (like if you write it on the spot).

Re: GitHub Copilot Chat Leaked Prompt

#406

Post by FrustratedMonky »

GTP wrote: post #396
FrustratedMonky wrote: post #326

Earlier quoted context omitted.

Wonder if anybody has used Godel's Incompleteness to prove this for our inner perception. If our brain is a calculation, then from inside the calculation, we can't prove ourselves to be real, right?

But we don't know for sure whether intelligence is computable or not.

Maybe that is the point, we can't prove it one way or the other, for human or machine. Can't prove a machine is conscious, and also can't prove we are. Maybe Gödel's theory could be used that it can't be done by humans. A human can't prove itself conscious because inside the human as system, can't prove all facts of the system.

Re: GitHub Copilot Chat Leaked Prompt

#407

Post by simonw »

wll wrote: post #401
simonw wrote: post #201

Earlier quoted context omitted.

I wrote about that here https://simonwillison.net/2022/Sep/17/prompt-injection-more-... - and talked about it here too: https://simonwillison.net/2023/May/2/prompt-injection-explai... Short version: you can't be 100% confident that AI based detection will catch every attack, because attackers are devious and can come up with increasingly unexpected and sneaky attacks. And where security is concerned, a 99% effective…

I disagree here. Just as it is impossible to perfectly secure a user-oriented operating system without severely limiting it (see Lockdown Mode), it might be impossible to prove injection-resistance in LLMs short of foundational advancements, but that doesn’t mean that we should dismiss attempts to mitigate with absolutism (I am referring to “none of our data would be safe anymore”), just as we don’t dismiss Apple for…

Would you trust you trust your private data to a system that was documented to fail to protect against 1/100 SQL injection vulnerabilities?

I wouldn't.

The difference between this and Apple releasing a security update is that when a traditional vulnerability is reported against an Apple product they can research the root cause of that vulnerability and produce a fix that they are certain is effective.

Prompt injection (currently) doesn't have fixes that work like that.

Re: GitHub Copilot Chat Leaked Prompt

#408

Post by yeck »

spacetime_cmplx wrote: post #88
perryizgr8 wrote: post #87

Earlier quoted context omitted.

It may not be the exact same model as GPT. They may have tweaked some parameters and almost definitely trained it on additional content relevant to the task of helping with coding. So you probably can't get the same output with just the same prompt.

Sure, in which case the real prompt is as useless as a hallucinated one, so what's the difference?

I don't think the purpose of getting the prompt leaked was to then use the prompt but just to expose the limitations of this approach to steering an LLM.

Re: GitHub Copilot Chat Leaked Prompt

#409

Post by scrollaway »

drstewart wrote: post #402
scrollaway wrote: post #392

Earlier quoted context omitted.

An example of poorly localised product which has real world consequences. Canva is a fantastic design software. If you type “business card” in templates, it has the standard us business card size but no European one. If you modify it to European sizes manually (closer to credit card size) it no longer recognises that this is a business card and no longer offers to print it on business card paper. In other words, desp…

Canva is based in Australia

And that changes what I said how?

Re: GitHub Copilot Chat Leaked Prompt

#410

Post by pmoriarty »

lkirk wrote: post #253
herpderperator wrote: post #92

Earlier quoted context omitted.

Is it actually able to base64 encode and decode completely correctly?

This thought led me to a strange conversation with chatgpt. I just base64 encoded "Hello chatgpt how are you" as "SGVsbG8gY2hhdGdwdCBob3cgYXJlIHlvdQ==" and sent it as the first prompt. Notice how there is no question mark in my phrase. Now here's my conversation with chatgpt: User SGVsbG8gY2hhdGdwdCBob3cgYXJlIHlvdQ== ChatGPT This appears to be a Base64 encoded string that translates to "Hello chattpt how are you?" Us…

From what I understand, these LLMs deliberately incorporate randomness. So it's not surprising at all that some of their answers would be imperfect.

Something else to consider is that these models work on tokens, not characters, so that might also account for why it made a mistake.

Post reply on HN