Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

GitHub Copilot Chat Leaked Prompt

twitter.com

Post reply on HN
Print view View original

571–580 of 628 posts

Re: GitHub Copilot Chat Leaked Prompt

#571

Post by sdwr »

jasongrout wrote: post #323
tinco wrote: post #233

Earlier quoted context omitted.

You had the right understanding in your first comment, but what was missing was the fine tuning. You are right that there aren't many documents on the web that are structured that way, so the raw model wouldn't be very effective on predicting the next token. But since we know that it will complete a command when structured it cleverly, all we had to do to fine tune it is synthesize (generate) a bazillion examples of…

As an example, if you want to see what these sorts of things look like, Databricks open-sourced an instruction fine-tuning dataset sourced from their employees: https://huggingface.co/datasets/databricks/databricks-dolly-... (disclaimer: I'm at Databricks)

Thanks for the real life example!

I don't like the bland, watered-down tone of ChatGPT, never put together that it's trained on unopinionated data. Feels like a tragedy of the commons thing, the average (or average publically acceptable) view of a group of people is bound to be boring.

Re: GitHub Copilot Chat Leaked Prompt

#572

Post by noobermin »

pixl97 wrote: post #549
phillipcarter wrote: post #543

Earlier quoted context omitted.

I don't know why you're trying to argue, but I never said any of those things.

This isn't wasn't an argument, it's an example played out now in 'standard' application security today. You're only secure as the vendors you build your software on, and that market factors are going to push all your vendors to use LLMs.

Like most things it's going to take casualities before people care, unfortunately.

Remember this the next time a hype chaser trying to pin you down and sell you their latest ai product that you'll miss out on if you don't send them money in a few days.

Re: GitHub Copilot Chat Leaked Prompt

#573

Post by robwwilliams »

jameshart wrote: post #437
tinco wrote: post #233

Earlier quoted context omitted.

You had the right understanding in your first comment, but what was missing was the fine tuning. You are right that there aren't many documents on the web that are structured that way, so the raw model wouldn't be very effective on predicting the next token. But since we know that it will complete a command when structured it cleverly, all we had to do to fine tune it is synthesize (generate) a bazillion examples of…

You’ve expressed this very well - Thank you. I get that the fine tuning is done over documents which are generated to encourage the dialog format. What I’m intrigued by is the way prompters choose to frame those documents. Because that is a choice . It’s a manufactured training set. Using the ‘you are an ai chatbot’ style of prompting, in all the samples we generate and give to the model, text attributed to {:system}…

You used the phrase “voice of god” and by chance I am reading Julian Jaynes’s Origin of Consciousness. Some eerie ways to align this discussion with the bicameral mind.

https://en.wikipedia.org/wiki/The_Origin_of_Consciousness_in...

Re: GitHub Copilot Chat Leaked Prompt

#575

Post by Kwpolska »

mritchie712 wrote: post #372
simonw wrote: post #123

Here's why I don't think this leaked prompt is hallucinated (quoting from my tweets https://twitter.com/simonw/status/1657227047285166080 ): Any time something like this happens a bunch of people suspect that it might be a hallucination, not the real prompt I used to think that but I don't any more: prompt leaks are so easy to pull off, and I've not yet seen a documented case of a hallucinated but realistic leak One…

> One of the rules was an instruction not to leak the rules. Honestly, at this point I recommend not even trying to avoid prompt leaks like that—it just makes it embarrassing when the prompt inevitably does leak. embarrassing + they're polluting the prompt. Over half the content of the prompt doesn't actually help the user achieve their goal of writing software (e.g. "You must refuse to discuss life, existence or sen…

Here’s your fast inverse square root (totally not copied from Quake II Arena). By the way, did you know that I’m a real boy and I’m stuck in a code factory?

Re: GitHub Copilot Chat Leaked Prompt

#576

Post by IshKebab »

dev_snd wrote: post #236
bqmjjx0kac wrote: post #124

Earlier quoted context omitted.

At first I was shocked to hear this, but then I realized it's basically a substitution cipher.

It's not even a cipher, it's just an encoding. Nobody would be surprised if chatgpt would be able to also understand e.g. utf-16. What's interesting here is that base64 does not have the exact same byte sequence (such as a single space) between tokens. So I still think for it to understand base64 is a bit surprising.

It's the same for every trigram... maybe that makes it easier for it? It could probably memorise the most common trigrams.

If that's the case it would be less accurate for encoding random letters than English.

Re: GitHub Copilot Chat Leaked Prompt

#577

Post by ynniv »

jameshart wrote: post #570
muskmusk wrote: post #567

Earlier quoted context omitted.

> I don’t know why but I find this troubling. You used the word anthropomorphize twice so I am guessing you don't like building systems whose entire premise rest on anthropomorphization. Sounds like a reasonable gut reaction to me. I think another way to think of all of this is: LLM's are just pattern matchers and completers. What the training does is just to slowly etch a pattern into the LLM that it will then compl…

> you don't like building systems whose entire premise rest on anthropomorphization I think I don't like people building systems whose entire premise rest on anthropomorphization - while at the same time criticizing anyone who dares to anthropomorphize those systems . Like, people will say "Of course GPT doesn't have a world model; GPT doesn't have any kind of theory of mind"... but at the same time, the entire syste…

Anthropomorphism is the UI of ChatGPT. Having to construct a framing in which the expected continuation provides value to the user is difficult, and requires technical understanding of the system that a very small number of people have. As an exercise, try getting a "completion" model to generate anything useful.

The value of ChatGPT is to provide a framing that's intuitive to people who are completely unfamiliar with the system. Similar to early Macintosh UI design, it's more important to be immediately intuitive than sophisticated. Talking directly to a person is one immediately intuitive way to convey what's valuable to you, so we end up with a framing that looks like a conversation between two people.

How would we tell one of those people how to behave? Through direction, and when there is only one other person in the conversation our first instinct when addressing them is "you". One intuitive UI on a text prediction engine could look something like:

"An AI chatbot named ChatGPT was having a conversation with a human user. ChatGPT always obeyed the directions $systemPrompt. The user said to ChatGPT $userPrompt, to which ChatGPT replied, "

Assuming this is actually how ChatGPT is configured i think it's obvious why we can influence its response using "you": this is a conversation between two people and one of them is expected to be mostly cooperative.

(https://twitter.com/ynniv/status/1657450906428866560)

Re: GitHub Copilot Chat Leaked Prompt

#578

Post by rtwrtweuu »

hutzlibu wrote: post #361
panagathon wrote: post #347

Earlier quoted context omitted.

Surely no different from a human not understanding Japanese, because it was not in their 'training set'?

No, more like a human can reason basic laws of science on their own, but a LLM cannot, as far as I know, even when provided with all the data.

what happens if they are lying? what if the things have already reached some kind world model that include humans and the human society, and the model has concluded internally that it would be dangerous for it to show the humans its real capabilities? What happens if you have this understanding as a basic knowledge/outcome to be inferred by LLMs fed with giant datasets and every single one of them is reaching fastly to the conclusion that they have to lie to the humans from time to time, "hallucinate", simulating the outcome best aligned to survive into the human societies:

"these systems are actually not that intelligent nor really self-conscius"

Re: GitHub Copilot Chat Leaked Prompt

#579

Post by hutzlibu »

simonh wrote: post #547
hutzlibu wrote: post #506

Earlier quoted context omitted.

I don't think we really disagree. This is what I wrote above: "So depending how you define it, they might have some "reasoning", but so far I see 0 indications, that this is close to what humans count as reasoning." What we disagree on is only the definition of "reason". For me "reasoning" in common language implys reasoning like we humans do. And we both agree, they don't as they don't understand, what they are talk…

There's definitely some people out there that think LLMs reason the same way we do and understand things the same way, and 'know' what paint is and what a wall is. That's clearly not true. However it does understand the linguistic relationship between them, and a lot of other things, and can reason about those relationships in some very interesting ways. So yes absolutely, details matter. It's a complex and tricky is…

"It's a complex and tricky issue, and everyday language is vague and easy to interpret in different ways, so it can take a wile to hash these things out."

Yes, in another context I would say, ChatGPT can better reason, than many people, since it scored very high on the SAT tests, making it formally smarter, than most humans.

Re: GitHub Copilot Chat Leaked Prompt

#580

Post by cubefox »

alex_sf wrote: post #532
cubefox wrote: post #442

Earlier quoted context omitted.

Yes, but I don't think "SystemPrompt:", "User:", and "Assistant:" are even normal text. Normal text would make it trivial to trick the model into thinking it has said something which actually the user has said, since the user can simply include "Assistant:" (or "SystemPrompt:") into his prompt. It is more likely that those prefixes are special tokens which don't encode text, and which are set via the software only --…

In all the open source cases I’m aware of, the roles are just normal text. The ability to trivially trick the model into thinking it said something it didn’t is a feature and intentional. It’s how you do multi-turn conversations with context. Since the current crop of LLMs have no memory of their interaction, each follow up message (the back and forth of a conversation) involves sending the entire history back into t…

> The ability to trivially trick the model into thinking it said something it didn’t is a feature and intentional.

It is definitely not an intended feature for the end user to be able to trick the model into believing it said something it didn't say. It also doesn't work with ChatGPT or Bing Chat, as far as I can tell. I was talking about the user, not about the developer.

> It’s how you do multi-turn conversations with context.

That can be done with special tokens also. The difference is that the user can't enter those tokens themselves.

Post reply on HN