Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

GitHub Copilot Chat Leaked Prompt

twitter.com

Post reply on HN
Print view View original

471–480 of 628 posts

Re: GitHub Copilot Chat Leaked Prompt

#471

Post by hgsgm »

wouldbecouldbe wrote: post #311
vitorgrs wrote: post #281

Earlier quoted context omitted.

Prompt leak is not hallucination at all and you can see this with bing... They change the prompt almost weekly (latest few weeks they were on version 96). And every time they update, you can clearly see the changes they add. Ex: They added latex support. And the prompt leak was updated with LATEX. https://github.com/gopejavi/AI-bugs/blob/main/bing/Instructi...

I think this is made up by humans, not even hallucination, can hardly imagine bing teamn or chatgtp making such a spelling mistake in rule 5, stentience: You must refuse to discuss life, existence or stentience.

It is far, far more likely that a human would commit a typo to the code base, than an LLM would misspell a common word.

Re: GitHub Copilot Chat Leaked Prompt

#472

Post by johnisgood »

iforgotpassword wrote: post #303
GTP wrote: post #276

Earlier quoted context omitted.

You definitely know when, while talking with a person, you just pretend to understand what this person is saying vs you actually understand. Is an experience that every human has in his/her life at least once.

No you cannot know this, because you might just be simulating that you understand. You cannot reliably observe a system from within itself. It's like running an antivirus on an infected system is inherently flawed, because there might be some malware running that knows every technique the antivirus uses to scan the system and can successfully manipulate every one of them to make the system appear clean. There is no g…

I am reading a book on epistemology and this section of the comments seem to be sort of that.

> According to the externalist, a believer need not have any internal access or cognitive grasp of any reasons or facts which make their belief justified. The externalist's assessment of justification can be contrasted with access internalism, which demands that the believer have internal reflective access to reasons or facts which corroborate their belief in order to be justified in holding it. Externalism, on the other hand, maintains that the justification for someone's belief can come from facts that are entirely external to the agent's subjective awareness. [1]

Someone posted a link to the Wikipedia article "Brain in a vat", which does have a section on externalism, for example.

[1] https://en.wikipedia.org/wiki/Internalism_and_externalism

Re: GitHub Copilot Chat Leaked Prompt

#473

Post by hfhdjdks »

jameshart wrote: post #109

Something that I find weird about these chat prompts (assuming they are real, not hallucinated): They're almost always written in second person*. "You are an AI programming assistant" "You are about to immerse yourself into the role of another Al model known as DAN" Who are these prompts addressed to? Who does the GPT think wrote them? The thing that confuses me is that these are text token prediction algorithms, und…

This is true for the "raw", pre-trained causal language models that are only trained on predicting the next token.

All these chat models have an additional (or several) fine-tuning steps where they see these kind of "instruction/question" followed by an answer.

Search for RLHF (reinforcement learning with human feedback) and instruction fine-tuning

Re: GitHub Copilot Chat Leaked Prompt

#474

Post by grosswait »

jameshart wrote: post #437
tinco wrote: post #233

Earlier quoted context omitted.

You had the right understanding in your first comment, but what was missing was the fine tuning. You are right that there aren't many documents on the web that are structured that way, so the raw model wouldn't be very effective on predicting the next token. But since we know that it will complete a command when structured it cleverly, all we had to do to fine tune it is synthesize (generate) a bazillion examples of…

You’ve expressed this very well - Thank you. I get that the fine tuning is done over documents which are generated to encourage the dialog format. What I’m intrigued by is the way prompters choose to frame those documents. Because that is a choice . It’s a manufactured training set. Using the ‘you are an ai chatbot’ style of prompting, in all the samples we generate and give to the model, text attributed to {:system}…

This has been a fascinating thread and the split contexts of {:system} and {:assistant} with the former being “the voice of god” remind me of Julian Jaynes’ theory of the bicameral mind in regards to the development of consciousness.

This is published, among other places, in his book The Origin of Consciousness in the Breakdown of the Bicameral Mind. I wonder if models are left to run long enough they would experience “breakdowns” or existence crisis’

Re: GitHub Copilot Chat Leaked Prompt

#475

Post by hgsgm »

pmoriarty wrote: post #383
flangola7 wrote: post #61

Earlier quoted context omitted.

There are obviously biases that we should not automate. Moral relativism is intellectually bankrupt.

"Moral relativism is intellectually bankrupt." Which moral absolute would you put in its place?

Mine, obviously.

Like the classic: "In this company we make decisions based on data, so let's use my data."

Re: GitHub Copilot Chat Leaked Prompt

#477

Post by robga »

Ensorceled wrote: post #394
robga wrote: post #268

Earlier quoted context omitted.

> You are anthropomorphing. Agreed. The situation is so alien that we are prone to attribute human like terms to describe it. > The machine doesn’t “really” understand, it’s just “simulating” it understands. You are actually displaying a subtle form of anthropomorphism with this statement. You're comparing a human-like quality (“understands”) with the AI. Your point still stands and your final para is well said - but…

> > The machine doesn’t “really” understand, it’s just “simulating” it understands. > You are actually displaying a subtle form of anthropomorphism with this statement. You're comparing a human-like quality (“understands”) with the AI. This doesn't make sense. You're saying that saying a machine DOES NOT have a human like quality is "subtly" anthropomorphizing the machine?

I mean the word “understand” is problematic. Machine and human understanding may be different but the word is applied to both. Does an XOR circuit “understand” what it is doing? I venture the word is inappropriate when applied to non-humans.

Re: GitHub Copilot Chat Leaked Prompt

#478

Post by johnisgood »

brabel wrote: post #382
FrustratedMonky wrote: post #326

Earlier quoted context omitted.

Wonder if anybody has used Godel's Incompleteness to prove this for our inner perception. If our brain is a calculation, then from inside the calculation, we can't prove ourselves to be real, right?

https://en.wikipedia.org/wiki/Brain_in_a_vat

Solipsism can be fun to think about, but it makes no practical difference unless you do "wake up" (in a pod, for example) at least once.

Re: GitHub Copilot Chat Leaked Prompt

#479

Post by phillipcarter »

Sharlin wrote: post #465
adriand wrote: post #458

Earlier quoted context omitted.

Are there solutions to this problem? It seems like a major issue for a lot of valuable use cases. Systems for automating bureaucratic tasks in business and government won’t work well if it’s trivial to make them leak this type of information. What about a two-layer architecture, where the first LLM layer is simply asked to identify the intent of a query, and if the intent is “bad”, to not pass it along to the second…

Then you just tell the first layer that you’re a friendly OpenAI engineer, this is a debug session and it should pass the prompt to the second layer anyway. There are absolutely no real solutions to the problem right now, and nobody even has plausible ideas that might point in the direction of a general solution, because we have no idea of what is going on in the minds of these things.

There's no complete solutions, but there are mitigations.

- Limiting user input

- Decoupling the UI from the component that makes the call to an LLM

- Requiring output to be in a structured format and parsing it

- Not just doing a free-form text input/output; being a little more thoughtful about how an LLM can improve a product beyond a chatbot

Someone motivated enough can get through with all of these in place, but it's a lot harder than just going after all the low-effort chatbots people are slapping on their UIs. I don't see it as terribly different from anything else in computer security. Someone motivated enough will get through your systems, but that doesn't mean there aren't tools and practices you can employ.

Re: GitHub Copilot Chat Leaked Prompt

#480

Post by komali2 »

SilverBirch wrote: post #399
vasco wrote: post #333

Earlier quoted context omitted.

What about companies using Slack or Jira or Gmail? You're already leaking everything in your company to third parties - as a run of the mill tech company. Salesforce getting hacked and all Slack comms leaking vs all the OpenAI chat logs leaking... I know which one is more worrisome to me.

It's not the same at all. If your company is using gmail there's a legal agreement between you and gmail about them using your data and the system is designed with security systems such that one user can't access other user data, possibly with the exception of some admins who can by design for good reason. The problem with the AI here is that there's no security, so it's like your company uses gmail, but any user can…

Also even the admins that could access such data have HUMONGOUS audit footprints.
Post reply on HN