Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

GitHub Copilot Chat Leaked Prompt

twitter.com

Post reply on HN
Print view View original

241–250 of 628 posts

Re: GitHub Copilot Chat Leaked Prompt

#241

Post by mrtranscendence »

ilrwbwrkhv wrote: post #220
htag wrote: post #117

Earlier quoted context omitted.

Sure, the prompt is bland. The interesting sauce is GPT4 cannot keep a secret. If you have a GPT4 powered user interface be sure not to load it with context you do not want directly leaking to the user.

anybody who uses gpt 4 or codex to do any of their programming or talk about sensitive data are not thinking things through and will end up leaking everything in their companies. i soon expect to see a ban on ai tools for many companies.

Maybe, but are you really expecting Microsoft/OpenAI to leak histories for Copilot or ChatGPT? It would be flatly suicidal. I guess those histories could be stolen, but many (most?) companies already trust Microsoft with a bunch of data and, maybe astonishingly, it’s largely been Ok.

Re: GitHub Copilot Chat Leaked Prompt

#242

Post by moonchrome »

jameshart wrote: post #149
tel wrote: post #133

Earlier quoted context omitted.

If you play with a "raw" model such as LLaMA you'll find what you suggest is true. These models do what you'd expect of a model that was trained to predict the next token. It's quite tricky to convince such a model to do what you want. You have to conceptualize it and then imagine an optimal prefix leading to the sort of output you've conceptualized. That said, people discovered some fairly general-purpose prefixes,…

Right. But who's the 'you' who's being addressed by the {:system} prompt? Who is the {:assistant} supposed to think the {:system} is? Why should the {:assistant} output tokens that make it do what the {:system} tells it to? After all, the {:user} doesn't. The {:system} doesn't provide any instructions for how the {:user} is supposed to behave, the {:user} tokens are chosen arbitrarily and don't match the probabilitie…

If I asked someone to continue out conversation thread - who are you and who am I ? Is it an existential nightmare ? The person completing just has to simulate two users.

Now if you're capable of that you are capable of completing the thread from a friendly AI assistant.

Re: GitHub Copilot Chat Leaked Prompt

#243

Post by mrtranscendence »

simonw wrote: post #123

Here's why I don't think this leaked prompt is hallucinated (quoting from my tweets https://twitter.com/simonw/status/1657227047285166080 ): Any time something like this happens a bunch of people suspect that it might be a hallucination, not the real prompt I used to think that but I don't any more: prompt leaks are so easy to pull off, and I've not yet seen a documented case of a hallucinated but realistic leak One…

> and I've not yet seen a documented case of a hallucinated but realistic leak

How would you know? As far as I know no company has come out and confirmed that any purportedly leaked prompts are genuine.

Re: GitHub Copilot Chat Leaked Prompt

#244

Post by renewiltord »

jameshart wrote: post #109

Something that I find weird about these chat prompts (assuming they are real, not hallucinated): They're almost always written in second person*. "You are an AI programming assistant" "You are about to immerse yourself into the role of another Al model known as DAN" Who are these prompts addressed to? Who does the GPT think wrote them? The thing that confuses me is that these are text token prediction algorithms, und…

It's really very obvious and all laid out in the instructgpt paper https://openai.com/research/instruction-following

Re: GitHub Copilot Chat Leaked Prompt

#245

Post by renewiltord »

naet wrote: post #19

I think that a lot of the limits placed on these models / chat services don't do much to remove underlying bias but rather attempt to obfuscate them from the general public. ChatGPT, Dall-e, etc all make assumptions about identity or politics but try to sidestep direct requests around those topics to appear more neutral... but the bias still exists in the model and affects the answers.

Nobody has time for this shit. They did what they had to so that the ne'er-do-wells who do nothing could have volume lowered to background noise and now we can get on with the business about being useful.

Re: GitHub Copilot Chat Leaked Prompt

#246

Post by avereveard »

simonw wrote: post #173
mvdtnz wrote: post #170

Earlier quoted context omitted.

You're committing the toupee fallacy[0]. You think you'd know the fake one if you saw it, and these are real because, well look at them, they must be real. > prompt leaks are so easy to pull off Only if you assume the prompt you're seeing is real. How do you know? [0] https://en.wiktionary.org/wiki/toupee_fallacy

I don't know for certain. But I've been messing around with prompt injection for six months now, so I've developed pretty good intuition about this stuff. A frustrating thing about LLMs is that often "intuition" is the best we get! And it's really hard to explain intuition to other people.

well you can get the probability map of generated tokens and do differential analysis on the output as you change the input. it's not just text in text out.

Re: GitHub Copilot Chat Leaked Prompt

#247

Post by wunderwuzzi23 »

simonw wrote: post #103

Huh, this is the least interesting thing I've written about prompt injection in the last few weeks, but the only one to make it to the Hacker News homepage. Better recent posts: - Delimiters won’t save you from prompt injection - https://simonwillison.net/2023/May/11/delimiters-wont-save-y... - talks about why telling a model to follow delimiters like ``` won't protect against prompt injection, despite that being men…

Delimiters are shown quite often as possible mitigations, but they do not work. I had the same observation when doing the Prompt Engineering class from OpenAI/DeepLearningAI.

Basically every example was vulnerable, and I made it a special challenge to perform an indirect prompt injection for each one of them. This led to interesting exploits such as JSON object injections, HTML injection and even XSS. Overwriting order prices with the OrderBot was also quite fun. :)

Here is a post and Notebook I used to learn/repro and experiment with these issues (incl. JSON Object injection and XSS): https://embracethered.com/blog/posts/2023/adversarial-prompt...

Also, an older post about data exfil for bots (with a Discord bot as an example): https://embracethered.com/blog/posts/2023/ai-injections-thre...

Re: GitHub Copilot Chat Leaked Prompt

#249

Post by iamflimflam1 »

hliyan wrote: post #234

A long time ago, I wrote a short story about a military AI that had the power to launch warheads, but needed to first justify its decisions to a second AI whose entire purpose was to act as checks-and-balances for the first. Can't we do something similar with these models? The output of the main model is fed into a second model (to which the end users have no access) trained on determining what can and cannot be expo…

I’ve played around with this idea a bit. It’s a very interesting experiment. You can have a “supervisor” that looks at the input and the output and judges how well the question has been answered. You can put this in a loop with the supervisor giving hints on how to improve the answers.

This is very similar to how things like auto GPT work.

Re: GitHub Copilot Chat Leaked Prompt

#250

Post by skissane »

mrtranscendence wrote: post #239
skissane wrote: post #225

Earlier quoted context omitted.

> Society does this every day and people think nothing of it if the hardware is a fentanyl lab or a uranium enrichment facility. There's a big difference between laws preventing you from doing those things in the privacy of your own home, and laws preventing you from entertaining or expressing certain ideas or beliefs in the privacy of your own home. Freedom of thought, freedom of belief, freedom of expression, are g…

Source code is a form of literature; software-being-executed is not. It might be permissible for me to write a program that steals credentials and holds them for ransom, but actually running it would not be. Thus I don’t see why it would be unbelievable that training or inference of large language models might, plausibly, be limited in some way by law.

> It might be permissible for me to write a program that steals credentials and holds them for ransom, but actually running it would not be.

Because that’s communicating with a third party, no longer limited to the privacy of your own home. Fraud, extortion, etc, go beyond the bounds of accepted free expression.

> Thus I don’t see why it would be unbelievable that training or inference of large language models might, plausibly, be limited in some way by law.

What’s the fundamental difference between training an AI to generate propaganda and generating it by hand? If freedom of expression protects the latter, why does it not protect the former too?

Post reply on HN