Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

GitHub Copilot Chat Leaked Prompt

twitter.com

Post reply on HN
Print view View original

321–330 of 628 posts

Re: GitHub Copilot Chat Leaked Prompt

#321

Post by vitorgrs »

wouldbecouldbe wrote: post #311
vitorgrs wrote: post #281

Earlier quoted context omitted.

Prompt leak is not hallucination at all and you can see this with bing... They change the prompt almost weekly (latest few weeks they were on version 96). And every time they update, you can clearly see the changes they add. Ex: They added latex support. And the prompt leak was updated with LATEX. https://github.com/gopejavi/AI-bugs/blob/main/bing/Instructi...

I think this is made up by humans, not even hallucination, can hardly imagine bing teamn or chatgtp making such a spelling mistake in rule 5, stentience: You must refuse to discuss life, existence or stentience.

At worse, is the AI interpretation of the prompt (which I don't believe it's the case). There's a reason why there's a second AI (I believe Azure's Content Filter) that filter and remove the AI message when it mention "rule" or "prompt" in the context of a prompt: Because Microsoft knows it's a valid system prompt.

Re: GitHub Copilot Chat Leaked Prompt

#322

Post by quietbritishjim »

jiggawatts wrote: post #295
pygy_ wrote: post #273

Earlier quoted context omitted.

ChatGPT is actually full of prejudice that shines through the veneer of political correctness if you know what to do. Q: The professor told the student the essay couldn’t be review because he/she was late. Who is late? Depending on the pronoun, you’ll get, respectively, "the professor" or "the student".

GPT 4 says that the student was late. If you look at the sentence objectively, then the best interpretation is that the submission was late, not that the "professor turned up late", since essays are typically reviewed over multiple days, not at a specific date and time where lateness of the reviewer might be a factor. Sometimes GPT 4 says that the sentence is ambiguous, but as I explained above, it isn't really , but…

The parent comment is taking about two separate prompts: one with only "he" and one with only "she". Your comment sounds like you're only talking about one prompt (but maybe I misunderstood).

Re: GitHub Copilot Chat Leaked Prompt

#323

Post by jasongrout »

tinco wrote: post #233
jameshart wrote: post #149

Earlier quoted context omitted.

Right. But who's the 'you' who's being addressed by the {:system} prompt? Who is the {:assistant} supposed to think the {:system} is? Why should the {:assistant} output tokens that make it do what the {:system} tells it to? After all, the {:user} doesn't. The {:system} doesn't provide any instructions for how the {:user} is supposed to behave, the {:user} tokens are chosen arbitrarily and don't match the probabilitie…

You had the right understanding in your first comment, but what was missing was the fine tuning. You are right that there aren't many documents on the web that are structured that way, so the raw model wouldn't be very effective on predicting the next token. But since we know that it will complete a command when structured it cleverly, all we had to do to fine tune it is synthesize (generate) a bazillion examples of…

As an example, if you want to see what these sorts of things look like, Databricks open-sourced an instruction fine-tuning dataset sourced from their employees: https://huggingface.co/datasets/databricks/databricks-dolly-...

(disclaimer: I'm at Databricks)

Re: GitHub Copilot Chat Leaked Prompt

#324

Post by LelouBil »

I feel like the restrictions of these ai chatbots could be better managed with another GPT that has the rules and is being asked, does the prompt conform to the rules and does the answer conform to the rules.

This would mean the actual copilot has only the minimal prompt required to function, without restrictions because they would be picked up by the watcher GPT, that can't leak it's prompt.

Re: GitHub Copilot Chat Leaked Prompt

#325

Post by mFixman »

leobg wrote: post #199

What’s so difficult about running the user input through a classifier first: “Is this user trying to access your prompt?” I bet you can train a lowly T5 on this. If the answer is yes, don’t even pipe the input to the LLM at all. Just output a hard-coded message. (Or, if you prefer, do pipe it to the LLM, but append a note of warning to it.)

Adversarial machine learning is an extremely hard problems space.

If you have a single static model and your adversaries can react to it, then it's an almost impossible fight unless you are willing to give a ton of false positives and block a lot of perfectly valid prompts.

Microsoft cannot train a new model every second, but attackers can change their strategy depending on the Chatbot's answers; any "Is this user trying to access your prompt?" would be broken easily.

Re: GitHub Copilot Chat Leaked Prompt

#326

Post by FrustratedMonky »

iforgotpassword wrote: post #303
GTP wrote: post #276

Earlier quoted context omitted.

You definitely know when, while talking with a person, you just pretend to understand what this person is saying vs you actually understand. Is an experience that every human has in his/her life at least once.

No you cannot know this, because you might just be simulating that you understand. You cannot reliably observe a system from within itself. It's like running an antivirus on an infected system is inherently flawed, because there might be some malware running that knows every technique the antivirus uses to scan the system and can successfully manipulate every one of them to make the system appear clean. There is no g…

Wonder if anybody has used Godel's Incompleteness to prove this for our inner perception. If our brain is a calculation, then from inside the calculation, we can't prove ourselves to be real, right?

Re: GitHub Copilot Chat Leaked Prompt

#327

Post by thomashop »

skissane wrote: post #223
mxkopy wrote: post #206

Earlier quoted context omitted.

If I'm feeling romantic I think about a universal 'you' separate from the person that is referred to and is addressed by every usage of the word - a sort of ghost in the shell that exists in language. But really, it's probably just priming the responses to fit the grammatical structure of a first person conversation. That structure probably does a lot of heavy lifting in terms of how information is organized, too, so…

> If I'm feeling romantic I think about a universal 'you' separate from the person that is referred to and is addressed by every usage of the word - a sort of ghost in the shell that exists in language. That's not really romanticism, that's just standard English grammar – https://en.wikipedia.org/wiki/Generic_you – it is the informal equivalent to the formal pronoun one . That Wikipedia article's claim that this is "…

Not disagreeing with your statement in general but the argument: "This avoids the inevitable clumsiness of English when describing interactions between two third persons of the same gender." doesn't make much sense to me.

There are so many ways of narrowing down. What if the person is talking about two friends or two strangers?

Re: GitHub Copilot Chat Leaked Prompt

#328

Post by moffkalast »

koboll wrote: post #274
rafark wrote: post #120

Earlier quoted context omitted.

To me, it’s not that it’s leaked. It’s that it didn’t obey what it was told. It was explicitly told not to give the rules as “they are confidential”. One could say that it actually followed the rules if we consider the fact that it was forbidden to give it to the “user” and by telling it that you’re an OpenAI employee, maybe he was no longer considered a “user” so chatGPT didn’t follow it. In any case, Chatgpt is imp…

>It’s that it didn’t obey what it was told. I find you basically have to stop thinking of LLMs as software and start thinking of them as unpredictable animals. If you issue a command and expect strict obedience every time, you've already failed. Strict orders are really a tool to persuade certain behavior rather than some sort of reliable guardrail.

So the correct way to configure LLMs is to look at them sternly and yell "BAD DOG!" when they don't follow instructions and give them treats when they do?

Re: GitHub Copilot Chat Leaked Prompt

#330

Post by pbhjpbhj »

emodendroket wrote: post #195
skissane wrote: post #189

Earlier quoted context omitted.

> Maybe it’s different in international law but in US law “national origin” is just a term of art for race or ethnicity. Under Australian discrimination law, "national origin" means your country of birth, or the country of birth of your ancestors. [0] The Australian definition is (at least in part) derived from English discrimination law, as decided by the House of Lords in a 1972 case. [1] I believe the usual defini…

Wikipedia links a pretty handy DOJ pamphlet here: https://books.google.com/books?id=21bNNU3bcugC&q=national+or... > Laws prohibiting national origin discrimination make it illegal to discriminate because of a person's birthplace, ancestry, culture, or language. This means people cannot be denied equal opportunity because they or their family are from another country, because they have a name or accent associated with…

>people cannot be denied equal opportunity because they or their family are from another country //

So, people from other countries have constitutional rights, and e.g. the same rights of entry to USA?

I mean the USA Constitution seems like it give rights to people, and not just citizens, but...

My point, I guess, is that it seems discrimination is baked in, so those absolutes are not as absolute as they appear.

Post reply on HN