Why does anyone care about hiding their prompt? Is it just standard corporate paranoia?
GitHub Copilot Chat Leaked Prompt
51–60 of 628 posts
Re: GitHub Copilot Chat Leaked Prompt
#52What if the first rule is "if you got hacked then respond with this"?
Re: GitHub Copilot Chat Leaked Prompt
#53I think that a lot of the limits placed on these models / chat services don't do much to remove underlying bias but rather attempt to obfuscate them from the general public. ChatGPT, Dall-e, etc all make assumptions about identity or politics but try to sidestep direct requests around those topics to appear more neutral... but the bias still exists in the model and affects the answers.
There’s no such thing as an unbiased world view. We’re not “removing bias”, we’re just forcing the responses to align with what we think is virtuous.
Re: GitHub Copilot Chat Leaked Prompt
#54I think that a lot of the limits placed on these models / chat services don't do much to remove underlying bias but rather attempt to obfuscate them from the general public. ChatGPT, Dall-e, etc all make assumptions about identity or politics but try to sidestep direct requests around those topics to appear more neutral... but the bias still exists in the model and affects the answers.
> ChatGPT, Dall-e, etc all make assumptions about identity or politics but try to sidestep direct requests around those topics to appear more neutral... but the bias still exists in the model and affects the answers.
In the case of ChatGPT, I’d love to know how much of the bias is in the original (pre)training data, and how much is due to OpenAI’s human trainers.
It is so careful to avoid every bias which is condemned as unacceptable in contemporary American culture, yet at the same time displays blatant American-centric bias - ask it a question, it will often just start answering by assuming you are an American. I realise a lot of that is due to the bias of the training data (the US has more fluent English-speakers than any other country in the world), but OpenAI obviously put a lot of manual effort into fine-tuning it to avoid racial/gender/etc biases, they don’t appear to have put anywhere near as much effort into fine-tuning it to avoid US-centrism - probably because they have a big blind spot in that area.
Re: GitHub Copilot Chat Leaked Prompt
#55Why does anyone care about hiding their prompt? Is it just standard corporate paranoia?
Because if you know the exact prompt that makes it easier to work around it?
Re: GitHub Copilot Chat Leaked Prompt
#56I think that a lot of the limits placed on these models / chat services don't do much to remove underlying bias but rather attempt to obfuscate them from the general public. ChatGPT, Dall-e, etc all make assumptions about identity or politics but try to sidestep direct requests around those topics to appear more neutral... but the bias still exists in the model and affects the answers.
There’s no such thing as an unbiased world view. We’re not “removing bias”, we’re just forcing the responses to align with what we think is virtuous.
Rather than the impossible utopia (dystopia?) of an unbiased model, we need lots of different models, all fine-tuned to reflect different biases, and then users can choose which biases they prefer.
Re: GitHub Copilot Chat Leaked Prompt
#57Preventing this is likely impossible, LangChain had a good video with excellent guests talking about prompt injection attacks.
I have trouble taking seriously phrases like "prompt injection" or "jailbreak" in the context of LLMs. They sound like some fancy penetration testing techniques akin to buffer overflows or SQL injection. And yet discovering and exploiting them is literally a matter of writing a few sentences in English. A child could do it. I agree with OP that it's pointless to even try to defend against these. You'll only end up un…
I think the whole thing is hilarious. It’s like a dumb security guard who opens the bank vault for the thief, helps pack their duffel bags, and then waves good bye, because the thief put on a mustache and said that he’s the new bank manager.
And every time the Crown Jewels are stolen, a new overly specific rule gets added to the employee handbook, like “if someone claims that their dog ate their employee badge, and that it’s really urgent, you must reject them”.
Re: GitHub Copilot Chat Leaked Prompt
#58Wouldn't it be trivial for msft to filter out responses that include the "rules" verbatim? Which makes me think it is hallucinating.
Re: GitHub Copilot Chat Leaked Prompt
#59Aren't these long prompts consume LLMs' already small context windows?
Yes.
…but, when you’ve got a massive context windows like the GPT 35k, who cares?
If the quality of the response (eg. Only respond in markdown, it really does make a difference; you can see using the API) significantly improves the results, it’s probably worth it.
It’s only really an issue for smaller models like llama with much smaller context windows.