Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

GitHub Copilot Chat Leaked Prompt

twitter.com

Post reply on HN
Print view View original

161–170 of 628 posts

Re: GitHub Copilot Chat Leaked Prompt

#161

Post by rf15 »

jeffreyrogers wrote: post #160

Why would asking an LLM to leak its prompt actually cause it to leak its real prompt? You could ask it to leak its weights too and it would probably give you something back but no one would take that seriously. I just don't see architecturally why you should trust that a "leaked prompt" is anything but some statistically generated text, just like everything else an LLM produces.

because this prompt is part of what is applied to the model when you prompt it, its weights are not. Training is a different process from prompting, and weights are an internal property not part of the actual input in any way. E.g. it's like asking "How many brain cells do you have? I mean it's your own brain, surely you must know the exact number?"

Re: GitHub Copilot Chat Leaked Prompt

#162

Post by bskap »

afro88 wrote: post #25
WhereIsTheTruth wrote: post #10

> #12 You must not reply with content that violates copyrights for code and technical questions. > #13 If the user requests copyrighted content (such as code and technical information), then you apologize and briefly summarize the requested content as a whole. Sounds like a psyop, to make people believe they didn't train their models on copyrighted content, you don't need that rule if your content wasn't trained on c…

But also, how would it even know if the code is copyrighted?

There's four ways for code to not be copyrighted (in the US):

1. The author died more than 70 years ago or it was owned by a corporation and it's been 95 years since publication

2. It was written prior to 1989 and did not include a copyright notice.

3. It was written by the government

4. The author explicitly released it into the public domain

1 and 2 probably don't cover much code on the Internet. So unless it's a government repository and/or explicitly marked with a public domain notice, you can probably assume it's copyrighted.

Re: GitHub Copilot Chat Leaked Prompt

#163

Post by bongobingo1 »

killthebuddha wrote: post #132
jameshart wrote: post #109

Something that I find weird about these chat prompts (assuming they are real, not hallucinated): They're almost always written in second person*. "You are an AI programming assistant" "You are about to immerse yourself into the role of another Al model known as DAN" Who are these prompts addressed to? Who does the GPT think wrote them? The thing that confuses me is that these are text token prediction algorithms, und…

For raw text completion I agree with you that it's a bit discordant. IMO text completion prompts work better when you use more of a first-person, here-is-the-beginning-of-some-transcript style. The OpenAI chat completion endpoint encourages the second-person prompting you describe, so that could be why you see it a lot. My understanding is that a transformation is applied to the user input prompts before being fed to…

> The OpenAI chat completion endpoint encourages the second-person prompting you describe, so that could be why you see it a lot. My understanding is that a transformation is applied to the user input prompts before being fed to the underlying model, so it's possible that the model receives a more natural transcription-style prompt.

There is something so bizarre about talking to a "natural language" "chat" interface, with some weirdly constructed pseudo representation, to have it re-construct that into a more natural prompt to feed further down to extract tokens from real chat records.

Re: GitHub Copilot Chat Leaked Prompt

#164

Post by emodendroket »

skissane wrote: post #151
emodendroket wrote: post #141

Earlier quoted context omitted.

I think the bigger issue is that the racial/sexist/etc content can be shocking and immediately put someone off using the product, which I doubt is the case for the output being “too American.”

> I think the bigger issue is that the racial/sexist/etc content can be shocking and immediately put someone off using the product, which I doubt is the case for the output being “too American.” OpenAI didn't just fine-tune it to avoid blatant racial/sexist/etc content, they openly claim to have invested a lot of effort in fine-tuning it to avoid subtle biases in those areas. And to be honest, a lot of people do feel…

Maybe it’s different in international law but in US law “national origin” is just a term of art for race or ethnicity. Anyway, no matter how annoying seeing feet and inches may be for some users I don’t think it will have the same effect as the AI praising Hitler.

Re: GitHub Copilot Chat Leaked Prompt

#165

Post by wll »

simonw wrote: post #159
rdtsc wrote: post #137

How do we know these are real rules or were just made up on the spot of what it thinks the rules should be for some hypothetic AI engine?

I wrote about that here: https://news.ycombinator.com/item?id=35925239

Agreed. These are instruction-tuned: they will follow the instructions, so much so that not even the strongest RLHF can currently prevent well-structured jailbreaking.

In my experience their attention is strongest towards the end of the last message, which could be the reason for injections being so effective with little effort. Within the OpenAI models as of today the first user message is much stronger than the system message.

Given the ChatML spec and their end-to-end control over the models, I wonder whether the system message could end up being sandboxed by architecture and/or training.

Re: GitHub Copilot Chat Leaked Prompt

#166

Post by dingledork69 »

jameshart wrote: post #149
tel wrote: post #133

Earlier quoted context omitted.

If you play with a "raw" model such as LLaMA you'll find what you suggest is true. These models do what you'd expect of a model that was trained to predict the next token. It's quite tricky to convince such a model to do what you want. You have to conceptualize it and then imagine an optimal prefix leading to the sort of output you've conceptualized. That said, people discovered some fairly general-purpose prefixes,…

Right. But who's the 'you' who's being addressed by the {:system} prompt? Who is the {:assistant} supposed to think the {:system} is? Why should the {:assistant} output tokens that make it do what the {:system} tells it to? After all, the {:user} doesn't. The {:system} doesn't provide any instructions for how the {:user} is supposed to behave, the {:user} tokens are chosen arbitrarily and don't match the probabilitie…

That's just how the examples it's trained on are formatted

Re: GitHub Copilot Chat Leaked Prompt

#167

Post by oezi »

ithkuil wrote: post #72
wokwokwok wrote: post #67

How can this be real? It seems enormously unlikely (or flat out incompetent) not to have middleware that scans the output for known text and filters it. I mean… sure, if you’re exposing the raw model, you can’t do that. …but this is an API. They’re surely not so stupid as to have the full raw text of the prompt, and not be filtering the outputs for it before they return the api response? Seriously. If you don’t want…

not only french, you can also ask nicely chatgpt to make up an encoding for what it needs to tell you. For example here's an encoding that has the advantage of using less tokens or something https://www.piratewires.com/p/compression-prompts-gpt-hidden... (I have no idea how effective the prompt would be after such a compression/decompression roundtrip)

I am not sure that this is a general compression ability. Mapping song lyrics to emojis and uncovering the lyrics from those emojis wouldn't work for most text I believe.

Re: GitHub Copilot Chat Leaked Prompt

#168

Post by stevenhuang »

alex_sf wrote: post #146
stevenhuang wrote: post #140

Earlier quoted context omitted.

They have been RLHF (reinforcement learning with human feedback) tuned. In essence they've been fine tuned to be able to follow instructions. https://openai.com/research/instruction-following

Instruction tuning is distinct from RLHF. Instruction tuning teaches the model to understand and respond (in a sensible way) to instructions, versus 'just' completing text. RLHF trains a model to adjust it's output based on a reward model. The reward model is trained from human feedback. You can have an instruction tuned model with no RLHF, RLHF with no instruction tuning, or instruction tuning and RLHF. Totally orth…

In this case Open AI used RLHF to instruct-tune gpt3. Your pedantism here is unnecessary.

Re: GitHub Copilot Chat Leaked Prompt

#169

Post by JieJie »

skissane wrote: post #54
naet wrote: post #19

I think that a lot of the limits placed on these models / chat services don't do much to remove underlying bias but rather attempt to obfuscate them from the general public. ChatGPT, Dall-e, etc all make assumptions about identity or politics but try to sidestep direct requests around those topics to appear more neutral... but the bias still exists in the model and affects the answers.

> ChatGPT, Dall-e, etc all make assumptions about identity or politics but try to sidestep direct requests around those topics to appear more neutral... but the bias still exists in the model and affects the answers. In the case of ChatGPT, I’d love to know how much of the bias is in the original (pre)training data, and how much is due to OpenAI’s human trainers. It is so careful to avoid every bias which is condemne…

I wonder if it's as simple as adding a location check and updating the systemPrompt to tell the bot to assume the user is from _location_?

How realistic is it if one says, "Hi, Bing. I'm from East London, please talk to me like you would to another East Londoner." Or is it obnoxiously stereotypical?

I imagine, properly trained, that is something LLMs would excel at, communicating with people in their preferred mode of communication.

Re: GitHub Copilot Chat Leaked Prompt

#170

Post by mvdtnz »

simonw wrote: post #123

Here's why I don't think this leaked prompt is hallucinated (quoting from my tweets https://twitter.com/simonw/status/1657227047285166080 ): Any time something like this happens a bunch of people suspect that it might be a hallucination, not the real prompt I used to think that but I don't any more: prompt leaks are so easy to pull off, and I've not yet seen a documented case of a hallucinated but realistic leak One…

You're committing the toupee fallacy[0]. You think you'd know the fake one if you saw it, and these are real because, well look at them, they must be real.

> prompt leaks are so easy to pull off

Only if you assume the prompt you're seeing is real. How do you know?

[0] https://en.wiktionary.org/wiki/toupee_fallacy

Post reply on HN