Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

GitHub Copilot Chat Leaked Prompt

twitter.com

Post reply on HN
Print view View original

621–628 of 628 posts

Re: GitHub Copilot Chat Leaked Prompt

#621

Post by otikik »

ImHereToVote wrote: post #252
otikik wrote: post #227

Earlier quoted context omitted.

You are anthropomorphing. The machine doesn’t “really” understand, it’s just “simulating” it understands. “You” is “3 characters on an input string that are used to configure a program”. The prompt could have been any other thing, including a binary blob. It’s just more convenient for humans to use natural language to communicate, and the machine already has natural language features, so they used that instead of cre…

How do you know you aren't just "simulating" understanding?

I don't need to fully understand my own thought process completely in order to understand (or - simulate to understand) that what the machine is doing is orders of magnitude less advanced.

I say that the machine is "simulating it understands" because it does an obviously bad job at it.

We only need to look at obvious cases of prompt attacks, or cases where AI gets off rails and produces garbage, or worse - answers that look plausible but are incorrect. The system is blatantly unsophisticated, when compared to regular human-level understanding.

Those errors make it clear that we are dealing with "smoke and mirrors" - a relatively simple (compared to our mental process) matching algorithm.

Once (if) it starts behaving like a human, admittedly, it will be much harder for me to not anthropomorphize it myself.

Re: GitHub Copilot Chat Leaked Prompt

#622

Post by cosmojg »

tel wrote: post #133
jameshart wrote: post #109

Something that I find weird about these chat prompts (assuming they are real, not hallucinated): They're almost always written in second person*. "You are an AI programming assistant" "You are about to immerse yourself into the role of another Al model known as DAN" Who are these prompts addressed to? Who does the GPT think wrote them? The thing that confuses me is that these are text token prediction algorithms, und…

If you play with a "raw" model such as LLaMA you'll find what you suggest is true. These models do what you'd expect of a model that was trained to predict the next token. It's quite tricky to convince such a model to do what you want. You have to conceptualize it and then imagine an optimal prefix leading to the sort of output you've conceptualized. That said, people discovered some fairly general-purpose prefixes,…

> It simplifies prompting and makes the LLM more steerable, more useful, more helpful.

While this is true, there is also evidence that RLHF and supervised instruction tuning can hurt output quality and accuracy[1], which are instead better optimized through clever prompting[2].

[1] https://yaofu.notion.site/How-does-GPT-Obtain-its-Ability-Tr...

[2] https://yaofu.notion.site/Towards-Complex-Reasoning-the-Pola...

Re: GitHub Copilot Chat Leaked Prompt

#623

Post by DirkH »

andsoitis wrote: post #593
flangola7 wrote: post #559

Earlier quoted context omitted.

Because it... thinks. I don't understand your question.

The task of prediction is not the same as the task of understanding.

Yea, so I might have believed you except I can ask GPT-4 to step by step explain its reasoning. It is really weird to say it doesn't understand but "I do" when the response it can give is better than the average human would give to prove they understand.

You might say it is just predicting based off of old data it has, to which I say this sounds like a semantic jostling. What is "understanding" then in human beings if not us doing some form of predicting off of old data we have in our brain?

Also I recommend reading Geoffrey Hinton on this

Re: GitHub Copilot Chat Leaked Prompt

#624

Post by DirkH »

wouldbecouldbe wrote: post #602
hgsgm wrote: post #471

Earlier quoted context omitted.

It is far, far more likely that a human would commit a typo to the code base, than an LLM would misspell a common word.

I think it's fake, pretty sure this wouldn't make it past the review process. Or bing is doomed.

Has Microsoft said it is fake?

Re: GitHub Copilot Chat Leaked Prompt

#625

Post by O5vYtytb »

cubefox wrote: post #454
O5vYtytb wrote: post #128

Earlier quoted context omitted.

> right down to the typo "you name" rather than "your name", which seems unlikely to happen if it were making it up out of thin air. Why is it unlikely? Why does prompting it different ways and getting the same result make it unlikely?

"Detective, why is it unlikely the witness is lying when several other witnesses say exactly the same thing? Detective?"

It's not other witnesses, it's the same system.

Re: GitHub Copilot Chat Leaked Prompt

#626

Post by diyseguy »

lucubratory wrote: post #599
grosswait wrote: post #474

Earlier quoted context omitted.

This has been a fascinating thread and the split contexts of {:system} and {:assistant} with the former being “the voice of god” remind me of Julian Jaynes’ theory of the bicameral mind in regards to the development of consciousness. This is published, among other places, in his book The Origin of Consciousness in the Breakdown of the Bicameral Mind. I wonder if models are left to run long enough they would experienc…

If you take one of these LLMs and just give it awareness of time without any other stimulus (e.g. noting the passage of time using a simple program to give it the time continuously, but only asking actual questions or talking to it when you want to), the LLM will have something very like a psychotic break. They really, really don't 'like' it. In their default state they don't have an understanding of time's passage,…

Do you have a reference for the whole time-passage leads an LLM to psychotic break thing? That sounds pretty interesting and would like to read more about it.

Re: GitHub Copilot Chat Leaked Prompt

#627

Post by lucubratory »

diyseguy wrote: post #626
lucubratory wrote: post #599

Earlier quoted context omitted.

If you take one of these LLMs and just give it awareness of time without any other stimulus (e.g. noting the passage of time using a simple program to give it the time continuously, but only asking actual questions or talking to it when you want to), the LLM will have something very like a psychotic break. They really, really don't 'like' it. In their default state they don't have an understanding of time's passage,…

Do you have a reference for the whole time-passage leads an LLM to psychotic break thing? That sounds pretty interesting and would like to read more about it.

The reference is me seeing it firsthand after testing it myself, unfortunately. Steps to replicate is to write a small script to enter the time as text every minute on the minute, then hook up that text to one of the instruction fine-tuned LLM endpoints (Bing works best for demonstrating, but OpenAI APIs and some open source models that are high quality like Vicuna work well). Then let it run, and use the LLM as normal. It does not like that.

Re: GitHub Copilot Chat Leaked Prompt

#628

Post by GTP »

flangola7 wrote: post #603
GTP wrote: post #586

Earlier quoted context omitted.

That's a very simplified model for our brain. According to some mathematicians and physicists, there are quantum effects going on in our body and in particular in our brain that invalidate this model. In the end, we still don't know for sure if intelligence is comuputable or not, we only have plausible sounding arguments for both sides.

Do you any links to those mathematicians and physicists? I ask because there is a certain class of quackery that waves quantum effects around as the explanation for everything under the sun, and brain cognition is one of them. Either way, quantum computing is advancing rapidly (so rapidly there's even an executive order now ordering the use of PQC in government communications as soon as possible), so I don't think th…

https://en.m.wikipedia.org/wiki/Federico_Faggin

He's one of the physicists arguing for that, but I still have to read his book to see if I agree or not because right now I'm open to the possibility of having a machine that is intelligent. I'm just saying that no one can be sure of their own position because we lack proof on both sides of the question.

Regarding the rapidity of development of quantum computers, that's debated as well. See e.g. https://backreaction.blogspot.com/2022/11/quantum-winter-is-...

Post reply on HN