Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

GitHub Copilot Chat Leaked Prompt

twitter.com

Post reply on HN
Print view View original

441–450 of 628 posts

Re: GitHub Copilot Chat Leaked Prompt

#441

Post by jameshart »

otikik wrote: post #227
jameshart wrote: post #149

Earlier quoted context omitted.

Right. But who's the 'you' who's being addressed by the {:system} prompt? Who is the {:assistant} supposed to think the {:system} is? Why should the {:assistant} output tokens that make it do what the {:system} tells it to? After all, the {:user} doesn't. The {:system} doesn't provide any instructions for how the {:user} is supposed to behave, the {:user} tokens are chosen arbitrarily and don't match the probabilitie…

You are anthropomorphing. The machine doesn’t “really” understand, it’s just “simulating” it understands. “You” is “3 characters on an input string that are used to configure a program”. The prompt could have been any other thing, including a binary blob. It’s just more convenient for humans to use natural language to communicate, and the machine already has natural language features, so they used that instead of cre…

I am not anthropomorphizing. The person who wrote the prompt using ‘you’ is. I’m interested in why they chose to do that.

Re: GitHub Copilot Chat Leaked Prompt

#442

Post by cubefox »

alex_sf wrote: post #135
jameshart wrote: post #109

Something that I find weird about these chat prompts (assuming they are real, not hallucinated): They're almost always written in second person*. "You are an AI programming assistant" "You are about to immerse yourself into the role of another Al model known as DAN" Who are these prompts addressed to? Who does the GPT think wrote them? The thing that confuses me is that these are text token prediction algorithms, und…

Responding to prompts like that are part of the 'instruction tuning' process. After an LLM is trained on a large dataset, it will do a decent job of completion , which acts like you describe. The next step is to further tune it with a specific format. You'll feed in examples like so: SystemPrompt: You are a rude AI. User: Hello there! Assistant: You're lame, go away. SystemPrompt: You are a pleasant AI. User: Hello t…

Yes, but I don't think "SystemPrompt:", "User:", and "Assistant:" are even normal text. Normal text would make it trivial to trick the model into thinking it has said something which actually the user has said, since the user can simply include "Assistant:" (or "SystemPrompt:") into his prompt.

It is more likely that those prefixes are special tokens which don't encode text, and which are set via the software only -- or via the model, when it is finished with what it wanted to say. Outputting a token corresponding to "User:" would automatically mark the end of its message, and the beginning of the user prompt. Though Bing Chat also has the ability to end the conversation altogether (no further user prompt possible), which must be another special token.

Re: GitHub Copilot Chat Leaked Prompt

#443

Post by dgacmu »

jameshart wrote: post #149
tel wrote: post #133

Earlier quoted context omitted.

If you play with a "raw" model such as LLaMA you'll find what you suggest is true. These models do what you'd expect of a model that was trained to predict the next token. It's quite tricky to convince such a model to do what you want. You have to conceptualize it and then imagine an optimal prefix leading to the sort of output you've conceptualized. That said, people discovered some fairly general-purpose prefixes,…

Right. But who's the 'you' who's being addressed by the {:system} prompt? Who is the {:assistant} supposed to think the {:system} is? Why should the {:assistant} output tokens that make it do what the {:system} tells it to? After all, the {:user} doesn't. The {:system} doesn't provide any instructions for how the {:user} is supposed to behave, the {:user} tokens are chosen arbitrarily and don't match the probabilitie…

There is no "you". There is a text stream that is being completed with maximum likelihood. One way to imagine it is that there are a lot of documents that have things like "if you are in a lightning storm, you should ..." And "if you are stuck debugging windows, you should reboot before throwing your computer out the window".

Starting the prompt with "you" instructions evidently helps get the token stream in the right part of the model space to generate output its users (here, the people who programmed copilot) are generally happy with, because there are a lot of training examples that make that "explicitly instructed" kind of text completion somewhat more accurate.

Re: GitHub Copilot Chat Leaked Prompt

#444

Post by hutzlibu »

ghusbands wrote: post #405
hutzlibu wrote: post #364

Earlier quoted context omitted.

Yeah, but maybe this exact example, is included in the trainig set?

It's unlikely, and you can come up with any number of variations of logic puzzle that are not in the training set and that get correct answers most of the time. Remember that the results aren't consistent and you may need to retry now and then. Or just give it a lump of code and change you want and see that it often successfully does so, even when there's no chance the code was in the training set (like if you write…

"Or just give it a lump of code and change you want and see that it often successfully does so, even when there's no chance the code was in the training set"

I did not claim (but my wording above might have been bad), it can only repeat word for word, what it has in the training set.

But I do claim, that it cannot solve anything, where there has not been enough similar examples before.

At least that has been my experience with it as a coding assistant and matches of what I understand of the inner workings.

Apart from that, is a automatic door doing reasoning, because it applies "reason" to the known conditions?

if (something on the IR sensor) openDoor()

I don't think so and neither are LLMs from what I have seen so far. That doesn't mean, I think that they are not useful, or that I rule out, that they could develope even consciousness.

Re: GitHub Copilot Chat Leaked Prompt

#445

Post by cubefox »

cloudking wrote: post #185
rsync wrote: post #154

Earlier quoted context omitted.

Yes, but that is your parents' point: "And what kind of documents exist that begin with someone saying 'you are X, here are a bunch of rules for how X behaves', followed by a ..." Where, your parent asks, are all these reams of texts written in this manner ?

It's not that "you are X" type text has to be explicitly in the training data, it's that the model weights interpret "you are X" as an instruction that a human would receive as an emergent behavior after digesting a ton of human written text.

The base model wouldn't do that though, it would just predict the most likely follow up, which could e.g. simply be more instructions. After instruction fine-tuning the model does no longer "predict" tokens in this way.

Re: GitHub Copilot Chat Leaked Prompt

#446

Post by wll »

simonw wrote: post #433
wll wrote: post #428

Earlier quoted context omitted.

I believe we can identify and fix attempts to evade detection. It is semantic and neuron-dependent and black box-like and therefore totally bonkers in feeling and iteration compared to what we are used to, but it works well enough considering we are at the earliest stages of advanced usage.

I don't think "works well enough" is good enough. We are currently starting to wire LLMs up as AI-enhanced personal assistants - with the goal of giving them access to our email, and the ability to take actions on our behalf. If we widely deploy these systems the incentives for attackers to figure out prompt injection attacks that get last any probability-based filters we are using will be enormous. An attacker only…

How do we determine how vulnerable a system is without seeing how it is implemented? That is, can you generalize LLM usage to all apps and determine that the entire field is exposed?

Obviously it is a problem for end users, but that’s more of an optics and trust angle, just as browser extensions are a privacy nightmare.

I am as worried as you are on this front, given that people appear to be using browser extensions liberally, but that doesn’t mean all apps will be equally vulnerable. It is as implementation-dependent as it is dependent on the nature of today’s models.

I appreciate your writing on the topic, it is both a great overview and a reminder to staying engaged while looking for mitigations and best practices.

Re: GitHub Copilot Chat Leaked Prompt

#448

Post by ericlewis »

simonw wrote: post #173
mvdtnz wrote: post #170

Earlier quoted context omitted.

You're committing the toupee fallacy[0]. You think you'd know the fake one if you saw it, and these are real because, well look at them, they must be real. > prompt leaks are so easy to pull off Only if you assume the prompt you're seeing is real. How do you know? [0] https://en.wiktionary.org/wiki/toupee_fallacy

I don't know for certain. But I've been messing around with prompt injection for six months now, so I've developed pretty good intuition about this stuff. A frustrating thing about LLMs is that often "intuition" is the best we get! And it's really hard to explain intuition to other people.

Interesting to see you say this, as I said the same to someone I know! It is a hard to explain intuition that comes from messing with these models for hours and hours. Perhaps we are intuiting the weights? :P
Post reply on HN