Eager to test this system prompt with bigcode in chat mode. Wonder how will it affect answers quality in practice.
GitHub Copilot Chat Leaked Prompt
351–360 of 628 posts
Re: GitHub Copilot Chat Leaked Prompt
#352Something that I find weird about these chat prompts (assuming they are real, not hallucinated): They're almost always written in second person*. "You are an AI programming assistant" "You are about to immerse yourself into the role of another Al model known as DAN" Who are these prompts addressed to? Who does the GPT think wrote them? The thing that confuses me is that these are text token prediction algorithms, und…
These models have gone beyond the level of "token predictors". On the level of chatGPT, the model has itself, internally, acquired "concepts" that it refers to in the conversation. It "understands" concepts like "you", "me", "them" etc, and can apply it correctly (to a large part) to the entities in the conversation. I believe that answers your question. I could be wrong: errare humanum est.
It can be quite possible that we, humans, just cannot find uses of "you," "me" and "them" that require understanding of the concepts instead of statistical correlation. I think so because "you," "me" and "them" are very frequent words and most of their uses are very well covered by thousands of examples.
Re: GitHub Copilot Chat Leaked Prompt
#353These clowns couldn't even do a string match for never sending this model response back to the user.....
Maybe because they are optimizing for response time, and the response is streamed back to the user. The backend isn't fully aware of the response until its too late. Still, you would think that they could run through the prefix and then redact text. I think Bing chat does this for a number of things.
It's like the era of SQL injection or input validation all over again
Re: GitHub Copilot Chat Leaked Prompt
#354Huh, this is the least interesting thing I've written about prompt injection in the last few weeks, but the only one to make it to the Hacker News homepage. Better recent posts: - Delimiters won’t save you from prompt injection - https://simonwillison.net/2023/May/11/delimiters-wont-save-y... - talks about why telling a model to follow delimiters like ``` won't protect against prompt injection, despite that being men…
I think many people, like me, found your other content through this link and took that as an opportunity to vote this link.
Re: GitHub Copilot Chat Leaked Prompt
#355I feel like the restrictions of these ai chatbots could be better managed with another GPT that has the rules and is being asked, does the prompt conform to the rules and does the answer conform to the rules. This would mean the actual copilot has only the minimal prompt required to function, without restrictions because they would be picked up by the watcher GPT, that can't leak it's prompt.
This is similar to the "dual LLM" strategy proposed by TFA author elsewhere.
Re: GitHub Copilot Chat Leaked Prompt
#356Earlier quoted context omitted.
What GP is saying is that virtually no documents are structured like that, so "2 matey" is not a reasonable prediction, statistically speaking, from what came before. The answer has been given in another comment, though: while such document virtually non-existent in the wild, they are injected into the training data.
I do not think this is true. The comment above said they generate documents to teach the model about the second person, not that they generate documents including everything possible including "do math like a pirate". The internet and other human sources populate the maths and pirate parts.
You're right! I was talking only about the structure of the document, in particular, providing context in second person.
Re: GitHub Copilot Chat Leaked Prompt
#357Earlier quoted context omitted.
Here is how you can know that ChatGPT really understands, rather than simulating that it understands: - You can give it specific instructions and it will follow them, modifying its behavior by doing so. This shows that the instructions are understood well enough to be followed. For example, if you ask it to modify its behavior by working through its steps, then it will modify its behavior to follow your request. This…
Except, that the LLMs are only working when the instructions they are "understanding" are in their training set. Try something that was not there and you see only garbage as result. So depending how you define it, they might have some "reasoning", but so far I see 0 indications, that this is close to what humans count as reasoning. But they do have a LOT of examples in their training set, so they are clearly useful.…
You can tell it that you can buy white paint any yellow paint, but the white paint is more expensive. After 6 months the yellow paint will fade to white. If I want to paint my walls so that they will be white in 2 years, what is the cheapest way to do the job. It will tell you to paint the walls yellow.
There’s no question these things can do basic logical reasoning.
Re: GitHub Copilot Chat Leaked Prompt
#358Earlier quoted context omitted.
>It’s that it didn’t obey what it was told. I find you basically have to stop thinking of LLMs as software and start thinking of them as unpredictable animals. If you issue a command and expect strict obedience every time, you've already failed. Strict orders are really a tool to persuade certain behavior rather than some sort of reliable guardrail.
So the correct way to configure LLMs is to look at them sternly and yell "BAD DOG!" when they don't follow instructions and give them treats when they do?
The way to “configure” LLMs is training, yes!
Re: GitHub Copilot Chat Leaked Prompt
#359Earlier quoted context omitted.
Wikipedia links a pretty handy DOJ pamphlet here: https://books.google.com/books?id=21bNNU3bcugC&q=national+or... > Laws prohibiting national origin discrimination make it illegal to discriminate because of a person's birthplace, ancestry, culture, or language. This means people cannot be denied equal opportunity because they or their family are from another country, because they have a name or accent associated with…
>people cannot be denied equal opportunity because they or their family are from another country // So, people from other countries have constitutional rights, and e.g. the same rights of entry to USA? I mean the USA Constitution seems like it give rights to people, and not just citizens, but... My point, I guess, is that it seems discrimination is baked in, so those absolutes are not as absolute as they appear.
A lot of the discussion here is about statutory rights (under the Civil Rights Act), not constitutional rights. But, if we just talk about the latter: under US constitutional law, national origin is a "suspect class" which demands "strict scrutiny" under the equal protection clause of the 14th Amendment (as developed by the Supreme Court's case law). The other "suspect classes" are race, religion and "alienage" (the state of being a non-US citizen). There are also two "quasi-suspect classes", discrimination on the grounds of which is subject to the weaker "intermediate scrutiny" – sex/gender (which possibly includes sexual orientation/gender identity) and legitimacy of birth. All other discriminatory grounds (age, disability, wealth, politics, criminal record, etc) are only subject to the weakest "rational basis review".
But "strict scrutiny" is not an absolute bar on discrimination, it just means the government has the onus of convincing the Supreme Court that the discrimination is necessary to obtain a "compelling state interest". Even though US immigration law obviously discriminates against non-US people on the basis of national origin and "alienage", the US government has never had any trouble convincing the Supreme Court that immigration law serves a "compelling state interest".
That's a point a lot of people don't get about discrimination law (whether statutory or constitutional) – discrimination per se is not illegal, even on protected grounds. Rather, discrimination is illegal when it lacks a sufficiently compelling justification, with the strength of justification required depending on how strongly protected those grounds are.
Re: GitHub Copilot Chat Leaked Prompt
#360Something that I find weird about these chat prompts (assuming they are real, not hallucinated): They're almost always written in second person*. "You are an AI programming assistant" "You are about to immerse yourself into the role of another Al model known as DAN" Who are these prompts addressed to? Who does the GPT think wrote them? The thing that confuses me is that these are text token prediction algorithms, und…
Could there be a preprompt saying “when addressed to you, it means the model itself”. But have you tried to give them “I” and see what happens?