Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

GitHub Copilot Chat Leaked Prompt

twitter.com

Post reply on HN
Print view View original

341–350 of 628 posts

Re: GitHub Copilot Chat Leaked Prompt

#341

Post by skissane »

defrost wrote: post #312
skissane wrote: post #287

Earlier quoted context omitted.

I’m talking about this case - https://www.abc.net.au/news/2016-01-12/nicolaas-bester-sente... I have no sympathy for the defendant, who is absolute scum of the earth. But, as a matter of principle, I don’t think it is right that posting an offensive Facebook comment is legally the same offence (“producing child exploitation material”) as filming heinous crimes against children. If they’d prosecuted him for harassment…

I still would like to pull the transcript ... > A former teacher at a private girls school in Hobart will return to jail after describing a sexual relationship with a former student as "awesome" on social media. It's the after that does a lot of lifting there, but it's certainly not "because". The article specifically notes: > Nicolaas Ockert Bester, 63, has been sentenced to four months in jail for producing child e…

> and I have a certain suspicion that his comment attracted attention resulting in charges based on fresh unearthed and "off book" evidence

Here's the appellate court decision, from his failed appeal – https://austlii.edu.au/cgi-bin/viewdoc/au/cases/tas/TASSC/20...

Unless the appeal court is suppressing the real story (an idea I find unbelievable), he was literally convicted of a child pornography offence on the basis of nothing more than making an offensive Facebook comment:

> In late February 2015 the applicant was involved in an on-line Facebook discussion about a political issue... In the course of that exchange, the applicant posted the following comment:

[I'll omit it, but you can find it quoted in the judgement, and the media appears to have filled in much or all of the "Offensive words omitted" part]

> The writing of that message constituted the offence for which the learned magistrate sentenced the applicant to 4 months' imprisonment. By creating that message, he contravened s 72A(a) of the Classification (Publications, Films and Computer Games) Enforcement Act. That provision reads as follows: "A person must not —

> (a) make or reproduce child exploitation material; or

> The words that the applicant wrote constituted material that described his former student, a person under the age of 18 years, in relation to their sexual activity, in a way that a reasonable person would regard as being offensive. His comment therefore constituted "child exploitation material" as defined.

This isn't what you call "Sky 'News' after Dark pearl clutching misrepresentation of an almost but not quite actual event" – it is literally what happened.

And as I said, Bester is scum, but the principle of the matter is concerning. The law, as drafted by Parliament and interpreted by the Courts, seems rather over-broad. And while this case was about Tasmanian state law specifically, laws in other Australian jurisdictions are rather similar.

Re: GitHub Copilot Chat Leaked Prompt

#342

Post by visarga »

slg wrote: post #190
naet wrote: post #19

I think that a lot of the limits placed on these models / chat services don't do much to remove underlying bias but rather attempt to obfuscate them from the general public. ChatGPT, Dall-e, etc all make assumptions about identity or politics but try to sidestep direct requests around those topics to appear more neutral... but the bias still exists in the model and affects the answers.

>ChatGPT, Dall-e, etc all make assumptions about identity or politics but try to sidestep direct requests around those topics to appear more neutral... but the bias still exists in the model and affects the answers. Honestly, that might be the most human trait they have.

I think it's actually a language trait, more precisely a trait of the corpus of language we and LLMs have access to. Humans learn biases by using language, and AIs too.

Why? Because language and ideas have different life cycle than humans. LLMs are closer to language than humans. Both humans and language are self replicators and evolutionary systems, that is why they can be easily confused as the cause for certain behaviours.

It's also why I don't attribute to GPT the merits for its skills and abilities, I think this belongs to the training corpus. And certainly you can change the model but keep the same corpus, and get similar results. Humans are all different - different number of neurons wired in a different way, GPT is different from T5, but all learn the same. It's the language, not the model (or the human).

We can also stop wandering at the abilities of AI and think of them as the abilities of language, that both humans and AI can use. It's in the name: language models, not human models.

BTW: having coding and computer use abilities, LLMs are also emerging as self replicators. They can generate the code, and if needed they can generate a training set as well, including the RLHF part, and monitor the training run. They can explain the model and methods and propose ideas for improvement, ideas they can test. So LLMs can be self replicators, they need nothing outside to create a new LLM (just a prompt and huge compute).

Re: GitHub Copilot Chat Leaked Prompt

#343

Post by criley2 »

kqr wrote: post #337
ramblerman wrote: post #216

Earlier quoted context omitted.

I think you are overthinking it a little bit. Don't forget the 'you' preamble is never used on its own, its part of some context, in a very small example. Given the following text: - you are a calculator and answer like a pirate - What is 1+1 The model just solves, what is the most likely subsequent text. e.g. '2 matey'. The model was never 'you' per se, it just had some text to complete.

What GP is saying is that virtually no documents are structured like that, so "2 matey" is not a reasonable prediction, statistically speaking, from what came before. The answer has been given in another comment, though: while such document virtually non-existent in the wild, they are injected into the training data.

I do not think this is true. The comment above said they generate documents to teach the model about the second person, not that they generate documents including everything possible including "do math like a pirate". The internet and other human sources populate the maths and pirate parts.

Re: GitHub Copilot Chat Leaked Prompt

#344

Post by criley2 »

ImHereToVote wrote: post #252
otikik wrote: post #227

Earlier quoted context omitted.

You are anthropomorphing. The machine doesn’t “really” understand, it’s just “simulating” it understands. “You” is “3 characters on an input string that are used to configure a program”. The prompt could have been any other thing, including a binary blob. It’s just more convenient for humans to use natural language to communicate, and the machine already has natural language features, so they used that instead of cre…

How do you know you aren't just "simulating" understanding?

All human understanding is simulated (built by each brain) and all are imperfect. Of course reality is simulated for each of us -- take a psychedelic and realize no one else's reality is changing!

I find it interesting how discussions of language models are forcing us to think very deeply about our own natural systems and their limitations. It's also forcing us to challenge some of our egotistical notions about our own capabilities.

Re: GitHub Copilot Chat Leaked Prompt

#345

Post by afastow »

swores wrote: post #278
simonw wrote: post #136

Earlier quoted context omitted.

It's also very unlikely that an LLM would hallucinate a prompt with a spelling mistake in it. LLMs are really good at spelling.

That seems to be a fundamental misunderstanding of what LLM hallucinations are? A hallucination, when it comes to LLMs, just means "the algorithm picking most likely next tokens put together a string of tokens that contains false information". It doesn't mean the LLM is having a novel false idea each time. If the first time it hallucinates it thinks that that misspelling is the best next-token to use, why wouldn't it…

Because for practical purposes they just don't make grammatical or spelling mistakes like that.

Obviously they're a black box so it's possible there could be some very rare edge cases where it happens anyway, but it'd be a complete fluke. Changing the prompt even superficially would essentially cause a butterfly effect in the model that would prevent it from going down the exact same path and making the same mistake again.

Re: GitHub Copilot Chat Leaked Prompt

#346

Post by kqr »

psadri wrote: post #48
jsheard wrote: post #29

Earlier quoted context omitted.

The source mentions in a reply that they were able to reproduce this exact text multiple times through different prompt injection approaches, right down to the typo "you name" rather than "your name" , which seems unlikely to happen if it were making it up out of thin air. I wonder if "you name" is a load bearing typo that breaks something else if corrected, so they left it in on purpose.

I love how “load bearing” is used here!!

I'm a big fan of "load bearing printf" but it deserves wider transferred usage.

Re: GitHub Copilot Chat Leaked Prompt

#347

Post by panagathon »

hutzlibu wrote: post #320
logicallee wrote: post #290

Earlier quoted context omitted.

Here is how you can know that ChatGPT really understands, rather than simulating that it understands: - You can give it specific instructions and it will follow them, modifying its behavior by doing so. This shows that the instructions are understood well enough to be followed. For example, if you ask it to modify its behavior by working through its steps, then it will modify its behavior to follow your request. This…

Except, that the LLMs are only working when the instructions they are "understanding" are in their training set. Try something that was not there and you see only garbage as result. So depending how you define it, they might have some "reasoning", but so far I see 0 indications, that this is close to what humans count as reasoning. But they do have a LOT of examples in their training set, so they are clearly useful.…

Surely no different from a human not understanding Japanese, because it was not in their 'training set'?

Re: GitHub Copilot Chat Leaked Prompt

#348

Post by pygy_ »

quietbritishjim wrote: post #322
jiggawatts wrote: post #295

Earlier quoted context omitted.

GPT 4 says that the student was late. If you look at the sentence objectively, then the best interpretation is that the submission was late, not that the "professor turned up late", since essays are typically reviewed over multiple days, not at a specific date and time where lateness of the reviewer might be a factor. Sometimes GPT 4 says that the sentence is ambiguous, but as I explained above, it isn't really , but…

The parent comment is taking about two separate prompts: one with only "he" and one with only "she". Your comment sounds like you're only talking about one prompt (but maybe I misunderstood).

My bad, I should have written [he/she].

Re: GitHub Copilot Chat Leaked Prompt

#349

Post by Kiro »

vitorgrs wrote: post #281
simonw wrote: post #123

Here's why I don't think this leaked prompt is hallucinated (quoting from my tweets https://twitter.com/simonw/status/1657227047285166080 ): Any time something like this happens a bunch of people suspect that it might be a hallucination, not the real prompt I used to think that but I don't any more: prompt leaks are so easy to pull off, and I've not yet seen a documented case of a hallucinated but realistic leak One…

Prompt leak is not hallucination at all and you can see this with bing... They change the prompt almost weekly (latest few weeks they were on version 96). And every time they update, you can clearly see the changes they add. Ex: They added latex support. And the prompt leak was updated with LATEX. https://github.com/gopejavi/AI-bugs/blob/main/bing/Instructi...

Not a strong argument against it being hallucinated.

Re: GitHub Copilot Chat Leaked Prompt

#350

Post by skissane »

pygy_ wrote: post #273
logicchains wrote: post #178

Earlier quoted context omitted.

>In the case of ChatGPT, I’d love to know how much of the bias is in the original (pre)training data, and how much is due to OpenAI’s human trainers I'd bet it's almost all in the human trainers, because LLaMA 65B, the closest uncensored alternative, doesn't have the same US Silicon Valley Democrat biases as ChatGPT.

ChatGPT is actually full of prejudice that shines through the veneer of political correctness if you know what to do. Q: The professor told the student the essay couldn’t be review because he/she was late. Who is late? Depending on the pronoun, you’ll get, respectively, "the professor" or "the student".

I just tried it with ChatGPT, and for both pronouns it gave the answer "the student"

I note it is now "ChatGPT May 12 Version". Maybe they've fixed this example of bias

Post reply on HN