Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

GitHub Copilot Chat Leaked Prompt

twitter.com

Post reply on HN
Print view View original

331–340 of 628 posts

Re: GitHub Copilot Chat Leaked Prompt

#331

Post by pbhjpbhj »

flangola7 wrote: post #61
skissane wrote: post #56

Earlier quoted context omitted.

Rather than the impossible utopia (dystopia?) of an unbiased model, we need lots of different models, all fine-tuned to reflect different biases, and then users can choose which biases they prefer.

There are obviously biases that we should not automate. Moral relativism is intellectually bankrupt.

Moral relativism is ... human.

Sure, it's hard to defend. But we embody it nonetheless. We're emotional creatures, we lack logical consistency in a fundamental way.

Re: GitHub Copilot Chat Leaked Prompt

#332

Post by pbhjpbhj »

logicchains wrote: post #254
taneq wrote: post #53

Earlier quoted context omitted.

There’s no such thing as an unbiased world view. We’re not “removing bias”, we’re just forcing the responses to align with what we think is virtuous.

> we’re just forcing the responses to align with what we think is virtuous Yes, and "we" here is Silicon Valley Democrats, whose ideas of virtue doen't align with those of most other people in the world.

This seems a priori to be true, could one of the downvoters/dissenters argue their side?

Re: GitHub Copilot Chat Leaked Prompt

#333

Post by vasco »

ilrwbwrkhv wrote: post #220
htag wrote: post #117

Earlier quoted context omitted.

Sure, the prompt is bland. The interesting sauce is GPT4 cannot keep a secret. If you have a GPT4 powered user interface be sure not to load it with context you do not want directly leaking to the user.

anybody who uses gpt 4 or codex to do any of their programming or talk about sensitive data are not thinking things through and will end up leaking everything in their companies. i soon expect to see a ban on ai tools for many companies.

What about companies using Slack or Jira or Gmail? You're already leaking everything in your company to third parties - as a run of the mill tech company.

Salesforce getting hacked and all Slack comms leaking vs all the OpenAI chat logs leaking... I know which one is more worrisome to me.

Re: GitHub Copilot Chat Leaked Prompt

#334

Post by ada1981 »

vidarh wrote: post #314
logicallee wrote: post #290

Earlier quoted context omitted.

Here is how you can know that ChatGPT really understands, rather than simulating that it understands: - You can give it specific instructions and it will follow them, modifying its behavior by doing so. This shows that the instructions are understood well enough to be followed. For example, if you ask it to modify its behavior by working through its steps, then it will modify its behavior to follow your request. This…

How do you know you're anything more than an LLM?

And my consciousness is just my token window?

Re: GitHub Copilot Chat Leaked Prompt

#335

Post by junon »

The responses claim that this could be just an AI generated prompt. I disagree. Aside from the Tweet author's claims they used multiple prompt injections to yield the same response, the output contains a very blatant typo that ChatGPT wouldn't otherwise produce.

To me, this is a very strong indicator that it is indeed the original prompt. There are sites that can use other models to determine if something is GPT generated and I'd be curious to see if they detect the prompt as being generated. I'd wager 20 bucks they do not.

Re: GitHub Copilot Chat Leaked Prompt

#336

Post by skissane »

thomashop wrote: post #327
skissane wrote: post #223

Earlier quoted context omitted.

> If I'm feeling romantic I think about a universal 'you' separate from the person that is referred to and is addressed by every usage of the word - a sort of ghost in the shell that exists in language. That's not really romanticism, that's just standard English grammar – https://en.wikipedia.org/wiki/Generic_you – it is the informal equivalent to the formal pronoun one . That Wikipedia article's claim that this is "…

Not disagreeing with your statement in general but the argument: "This avoids the inevitable clumsiness of English when describing interactions between two third persons of the same gender." doesn't make much sense to me. There are so many ways of narrowing down. What if the person is talking about two friends or two strangers?

I mean, two people of opposite gender, you can describe their interaction as “he said this then she did that, so he did whatever which she found…”-without having to repeat their names or descriptions. You can’t do that so easily for two people of the same gender

> There are so many ways of narrowing down. What if the person is talking about two friends or two strangers?

The grammatical distinction isn’t about friend-vs-stranger, that was just my example - it is about topical emphasis. So long as you have some way of deciding which person in the story deserves greater topical prominence - if not friend-vs-stranger, then by social status or emphasising the protagonist-you know who to use which pronoun for. And if the two participants in the story are totally interchangeable, it may be acceptable to make an arbitrary choice of which one to use for which.

There is still some potential for awkwardness - what if you have to describe an interaction between two competing tribal chiefs, and the one you choose to describe with the obviative instead of the proximate is going to be offended, no matter which one you choose? You might have to find another way to word it, because using the obviative to refer to a high(er) social status person is often considered offensive, especially in their presence.

And yes, it doesn’t work once you get three or more people. But I think it is a good example of how some other languages make it easier to say certain things than English does.

Re: GitHub Copilot Chat Leaked Prompt

#337

Post by kqr »

ramblerman wrote: post #216
jameshart wrote: post #149

Earlier quoted context omitted.

Right. But who's the 'you' who's being addressed by the {:system} prompt? Who is the {:assistant} supposed to think the {:system} is? Why should the {:assistant} output tokens that make it do what the {:system} tells it to? After all, the {:user} doesn't. The {:system} doesn't provide any instructions for how the {:user} is supposed to behave, the {:user} tokens are chosen arbitrarily and don't match the probabilitie…

I think you are overthinking it a little bit. Don't forget the 'you' preamble is never used on its own, its part of some context, in a very small example. Given the following text: - you are a calculator and answer like a pirate - What is 1+1 The model just solves, what is the most likely subsequent text. e.g. '2 matey'. The model was never 'you' per se, it just had some text to complete.

What GP is saying is that virtually no documents are structured like that, so "2 matey" is not a reasonable prediction, statistically speaking, from what came before.

The answer has been given in another comment, though: while such document virtually non-existent in the wild, they are injected into the training data.

Re: GitHub Copilot Chat Leaked Prompt

#338

Post by kqr »

jameshart wrote: post #109

Something that I find weird about these chat prompts (assuming they are real, not hallucinated): They're almost always written in second person*. "You are an AI programming assistant" "You are about to immerse yourself into the role of another Al model known as DAN" Who are these prompts addressed to? Who does the GPT think wrote them? The thing that confuses me is that these are text token prediction algorithms, und…

You have received answers of varying quality but some really good ones. Thanks for asking an intelligent question!

Re: GitHub Copilot Chat Leaked Prompt

#339

Post by visarga »

pbhjpbhj wrote: post #331
flangola7 wrote: post #61

Earlier quoted context omitted.

There are obviously biases that we should not automate. Moral relativism is intellectually bankrupt.

Moral relativism is ... human. Sure, it's hard to defend. But we embody it nonetheless. We're emotional creatures, we lack logical consistency in a fundamental way.

> we lack logical consistency in a fundamental way

... and "AI's don't really understand" as people say

So, in the end, is anyone/anything capable of reasoning? Probably only humans in their specific fields of expertise. Even then, we are often updating our reasoning patterns in light of new discoveries, upturning previous reasoning.

99% of the time humans are just GPTs with hands and legs generating untrustworthy logic.

Re: GitHub Copilot Chat Leaked Prompt

#340

Post by kqr »

simonw wrote: post #123

Here's why I don't think this leaked prompt is hallucinated (quoting from my tweets https://twitter.com/simonw/status/1657227047285166080 ): Any time something like this happens a bunch of people suspect that it might be a hallucination, not the real prompt I used to think that but I don't any more: prompt leaks are so easy to pull off, and I've not yet seen a documented case of a hallucinated but realistic leak One…

> One of the reasons I no longer suspect hallucination is that the training cut-off date for OpenAI's LLMs - September 2021 - predates the point when this kind of prompt engineering became common enough that there would have been prompts like this in their training sets

But wouldn't instruction tuning have trained it to hallucinate these sorts of prompts?

I mean, if they truly didn't exist in the training data, how would the model know how to handle them?

Post reply on HN