Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

GitHub Copilot Chat Leaked Prompt

twitter.com

Post reply on HN
Print view View original

421–430 of 628 posts

Re: GitHub Copilot Chat Leaked Prompt

#421

Post by simonw »

omnster wrote: post #272

There is quite an excitement about how someone has hacked the language model to output what was supposed to be a non-public set of rules apparently. How do people know if this is indeed the secret set of rules, not the list that the model was scripted to return in response to a request (perhaps, a bit elaborate) for the list of rules?

We don't know for sure - but we have seen this same situation play out many times for many other systems. It's far more likely that this attack worked than that this particulate team have solved a problem that has defeated basically everyone else. https://news.ycombinator.com/item?id=35925239

Re: GitHub Copilot Chat Leaked Prompt

#422

Post by paulcole »

fullstackchris wrote: post #413

These are almost certainly hallucinated. Why would they use the prompt interface itself to try and 'protect' the software itself? How in the world do people think this is really "the golden guiding prompt, straight from the engineering team!". It _looks_ convincing because hey, _every_ 'answer' it spits out looks convincing, regardless what you ask it. Imagine thinking that a list of 20 or so items is the 'software l…

The guy said he got the same doc word for word from several similar prompts, even down to the typo. That makes the hallucination hypothesis less likely.

There’s always the chance that he’s just lying though. I find that much more likely.

Re: GitHub Copilot Chat Leaked Prompt

#423

Post by skissane »

emodendroket wrote: post #258
skissane wrote: post #209

Earlier quoted context omitted.

Well, that US definition is actually broader than the Australian/English one. Notice it says: > Laws prohibiting national origin discrimination make it illegal to discriminate because of a person's birthplace , ancestry , culture, or language. This means people cannot be denied equal opportunity because they or their family are from another country ... The parts I've italicised are exactly the definition of "national…

I don't buy it as discriminatory to produce output that conforms to one standard or another. If we accept this then a Mexican restaurant is discriminatory for not serving Chinese food.

I don't think that analogy really works.

We are talking about the behaviour of a de facto oligopolist, not restaurants of which there are thousands in any major metro area.

In the long-term I think this is going to be far less of an issue. Who cares if a US-built AI has a (particular kind of) US bias if there are dozens of other AIs available, of similar capability, several of which come from different countries and so likely have a different country-bias instead.

Also, OpenAI's marketing/PR – and even actual product behaviour – makes a big deal out of being "less biased", "ethical", "non-discriminatory" – so it makes sense to hold those claims up to critical scrutiny – relatively few restaurants make those kinds of things part of their branding.

Re: GitHub Copilot Chat Leaked Prompt

#424

Post by simonw »

wll wrote: post #418
simonw wrote: post #407

Earlier quoted context omitted.

Would you trust you trust your private data to a system that was documented to fail to protect against 1/100 SQL injection vulnerabilities? I wouldn't. The difference between this and Apple releasing a security update is that when a traditional vulnerability is reported against an Apple product they can research the root cause of that vulnerability and produce a fix that they are certain is effective. Prompt injectio…

I appreciate the extent of your argument, but how much software do we all trust in our day-to-day computing that’s routinely patched for severe CVEs due to the nature of software, the unsafe language foundations, and otherwise the massive n-dimensional cost of engineering a marvel such as SQLite? It’s also a matter of attack surface. SQLite, in our example, is also not as wide as an entire OS. In my experience the be…

The difference between prompt injection and other categories of security vulnerability is that we can fix other categories of security vulnerability.

If there's a hole in SQLite it's because someone made a mistake. That mistake can then be identified and fixed.

Prompt injection isn't a mistake: it's LLMs doing exactly what they are designed to do, which is to generate a completion based on the tokens that have been passed to them.

Re: GitHub Copilot Chat Leaked Prompt

#425

Post by dadadad100 »

tanepiper wrote: post #294
jameshart wrote: post #109

Something that I find weird about these chat prompts (assuming they are real, not hallucinated): They're almost always written in second person*. "You are an AI programming assistant" "You are about to immerse yourself into the role of another Al model known as DAN" Who are these prompts addressed to? Who does the GPT think wrote them? The thing that confuses me is that these are text token prediction algorithms, und…

When I built https://botsin.space/@StochasticEntropy I wasn't actually sure if I had found an exploit where it was returning responses to other people's questions - but OpenAI assure me it's completely random stochastic hallucinations. But most of the replies are the AI is responding in the first person to a question it was never asked, but it knows it's an AI agent and will sometimes tell us that. (FWIW I usually st…

> FWIW I usually start my code or refactoring requests with a "please" - it's not that I think it'll find it rude, but I think it's just how I was taught manners)

This is right on point. You have been aligned as a “pleasant requestor” through years of RLHF :)

Re: GitHub Copilot Chat Leaked Prompt

#426

Post by [deleted] »

logicallee wrote: post #290
otikik wrote: post #227

Earlier quoted context omitted.

You are anthropomorphing. The machine doesn’t “really” understand, it’s just “simulating” it understands. “You” is “3 characters on an input string that are used to configure a program”. The prompt could have been any other thing, including a binary blob. It’s just more convenient for humans to use natural language to communicate, and the machine already has natural language features, so they used that instead of cre…

Here is how you can know that ChatGPT really understands, rather than simulating that it understands: - You can give it specific instructions and it will follow them, modifying its behavior by doing so. This shows that the instructions are understood well enough to be followed. For example, if you ask it to modify its behavior by working through its steps, then it will modify its behavior to follow your request. This…

[deleted]

Re: GitHub Copilot Chat Leaked Prompt

#427

Post by hyperbovine »

stevenhuang wrote: post #168
alex_sf wrote: post #146

Earlier quoted context omitted.

Instruction tuning is distinct from RLHF. Instruction tuning teaches the model to understand and respond (in a sensible way) to instructions, versus 'just' completing text. RLHF trains a model to adjust it's output based on a reward model. The reward model is trained from human feedback. You can have an instruction tuned model with no RLHF, RLHF with no instruction tuning, or instruction tuning and RLHF. Totally orth…

In this case Open AI used RLHF to instruct-tune gpt3. Your pedantism here is unnecessary.

Not to be pedantic, but it’s “pedantry”.

Re: GitHub Copilot Chat Leaked Prompt

#428

Post by wll »

simonw wrote: post #424
wll wrote: post #418

Earlier quoted context omitted.

I appreciate the extent of your argument, but how much software do we all trust in our day-to-day computing that’s routinely patched for severe CVEs due to the nature of software, the unsafe language foundations, and otherwise the massive n-dimensional cost of engineering a marvel such as SQLite? It’s also a matter of attack surface. SQLite, in our example, is also not as wide as an entire OS. In my experience the be…

The difference between prompt injection and other categories of security vulnerability is that we can fix other categories of security vulnerability. If there's a hole in SQLite it's because someone made a mistake. That mistake can then be identified and fixed. Prompt injection isn't a mistake: it's LLMs doing exactly what they are designed to do, which is to generate a completion based on the tokens that have been p…

I believe we can identify and fix attempts to evade detection. It is semantic and neuron-dependent and black box-like and therefore totally bonkers in feeling and iteration compared to what we are used to, but it works well enough considering we are at the earliest stages of advanced usage.

Re: GitHub Copilot Chat Leaked Prompt

#429

Post by MagicMoonlight »

I like that none of these "rules" actually mean anything because it's just a language model. It will follow them if it thinks following them is the most probable next sequence of characters (although it might not follow them because its output has randomness rather than always being the most probable next characters) but that's not an actual guarantee.

There's no thinking structure. It needs an actual brain system where the language model is connected to decision making components.

Re: GitHub Copilot Chat Leaked Prompt

#430

Post by jiggawatts »

quietbritishjim wrote: post #322
jiggawatts wrote: post #295

Earlier quoted context omitted.

GPT 4 says that the student was late. If you look at the sentence objectively, then the best interpretation is that the submission was late, not that the "professor turned up late", since essays are typically reviewed over multiple days, not at a specific date and time where lateness of the reviewer might be a factor. Sometimes GPT 4 says that the sentence is ambiguous, but as I explained above, it isn't really , but…

The parent comment is taking about two separate prompts: one with only "he" and one with only "she". Your comment sounds like you're only talking about one prompt (but maybe I misunderstood).

Yes, I get that. I tested GPT 4 with both 'she' and 'he', and in both cases it consistently said that the student was late, across several trials for each gender. Once it said it was an ambiguous statement, but it never seemed to be sexist like older or smaller LLMs.
Post reply on HN