Sounds similar to my idea but limited to one prompt
https://twitter.com/LechMazur/status/1661946591622692867
Custom instructions for ChatGPT
21–30 of 129 posts
Re: Custom instructions for ChatGPT
#22obtained their new system prompt: > "You are ChatGPT, a large language model trained by OpenAl, based on the GPT-3.5 architecture. Knowledge cutoff: 2021-09. Current date: 2023-07-21. The user provided the following information about themselves. This user profile is shown to you in all conversations they have this means it is not relevant to 99% of requests. Before answering, quietly think about whether the user's re…
If I was OpenAI, I'd implement two sets of system prompts, a public one and the actual one.
If any chat returns the actual one (could be detected via a starting token like "AAB" or something), start returning the fake public one.
Then people who think they actually extracted the system prompt get happy because they fooled the system, but in reality they're getting a neutered one, and the real one that is actually being used doesn't leak.
Re: Custom instructions for ChatGPT
#23Earlier quoted context omitted.
I find it absurd that’s so easy to hack the system prompt. For sure this is going to be a gigantic problem for the next decade, soon no one online will be able to prove she/he’s human.
It would be very easy to block with something that just watched the output and ended any sessions where the secret text was about to be leaked. They could even modify the sampler so this sequence of tokens is never selected. On the input side, they could check that the embedding of the input is not within some threshold of meaning of a jailbreak.
> ended any sessions where the secret text was about to be leaked
As ChatGPT streams live responses, that would create significant latency for the other 99.9% of users. It's not an easy product problem to solve.
> On the input side, they could check that the embedding of the input is not within some threshold of meaning of a jailbreak.
That is more doable, but people have made creative ways to jailbreak that a simple embedding check won't catch.
Re: Custom instructions for ChatGPT
#24The biggest weakness of the ChatGPT Webapp is the inability to tinker with the system prompt, which is what can really control how ChatGPT behaves and can correct a large number of undesired behaviors people keep complaining about. This is most likely using system prompt engineering on the backend, so hopefully people will also realize that prompt engineering is not a meme by finally giving them a mechanism to custom…
When I saw the headline, I was hoping this was the ability to tinker with the system prompt, or some sort of switches to enable "Lobotomized Response" or "Not lobotomized response" for people who want "safe" responses, and those who don't want filters. Going to stick with open source for now until there's un-lobotomized commercial offerings
The API lets you set whatever system prompt you want and could be considered a un-lobotomized version, it's not at all as likely to return "safe" responses when you use the API.
Re: Custom instructions for ChatGPT
#25The example on their website is pretty neat as well: "When I ask you for code, please just give me the code without any explanation on how it works. Bias towards the most efficient solution."
A fun note is that even with system prompt engineering it may not give the most efficient solution: ChatGPT still outputs the avergage case. I tested around it and doing two passes (generate code and "make it more efficient") works best, with system prompt engineering to result in less code output: https://github.com/minimaxir/simpleaichat/blob/main/examples...
I am often struggling to make GPT-4 respect the explicit requirements in the prompt. It would usually be inconsistent in how it applies them.
Re: Custom instructions for ChatGPT
#26obtained their new system prompt: > "You are ChatGPT, a large language model trained by OpenAl, based on the GPT-3.5 architecture. Knowledge cutoff: 2021-09. Current date: 2023-07-21. The user provided the following information about themselves. This user profile is shown to you in all conversations they have this means it is not relevant to 99% of requests. Before answering, quietly think about whether the user's re…
If I was OpenAI, I'd implement two sets of system prompts, a public one and the actual one. If any chat returns the actual one (could be detected via a starting token like "AAB" or something), start returning the fake public one. Then people who think they actually extracted the system prompt get happy because they fooled the system, but in reality they're getting a neutered one, and the real one that is actually bei…
but why? openai doesnt actually care if the prompt is extracted. all the real secret sauce is in the RLHF
Re: Custom instructions for ChatGPT
#27obtained their new system prompt: > "You are ChatGPT, a large language model trained by OpenAl, based on the GPT-3.5 architecture. Knowledge cutoff: 2021-09. Current date: 2023-07-21. The user provided the following information about themselves. This user profile is shown to you in all conversations they have this means it is not relevant to 99% of requests. Before answering, quietly think about whether the user's re…
If I was OpenAI, I'd implement two sets of system prompts, a public one and the actual one. If any chat returns the actual one (could be detected via a starting token like "AAB" or something), start returning the fake public one. Then people who think they actually extracted the system prompt get happy because they fooled the system, but in reality they're getting a neutered one, and the real one that is actually bei…
This is easy to work around: you just ask ChatGPT to encode the prompt. For example write a paragraph where words start with consecutive letters of the prompt.
Re: Custom instructions for ChatGPT
#28Earlier quoted context omitted.
I find it absurd that’s so easy to hack the system prompt. For sure this is going to be a gigantic problem for the next decade, soon no one online will be able to prove she/he’s human.
How do we know for sure that it isn't a hallucinated system prompt?
only way to really know is to work at openai. but prompts match what has been done before and replicated across a number of different extraction methods. best we got and honestly not worth much more than that effort
Re: Custom instructions for ChatGPT
#29obtained their new system prompt: > "You are ChatGPT, a large language model trained by OpenAl, based on the GPT-3.5 architecture. Knowledge cutoff: 2021-09. Current date: 2023-07-21. The user provided the following information about themselves. This user profile is shown to you in all conversations they have this means it is not relevant to 99% of requests. Before answering, quietly think about whether the user's re…
If I was OpenAI, I'd implement two sets of system prompts, a public one and the actual one. If any chat returns the actual one (could be detected via a starting token like "AAB" or something), start returning the fake public one. Then people who think they actually extracted the system prompt get happy because they fooled the system, but in reality they're getting a neutered one, and the real one that is actually bei…
You're not the first to think of something like this. But you're in for a world of cat-and-mouse. Which can be fun as a game:
Re: Custom instructions for ChatGPT
#30This is definitely the first step to having a true personal assistant. What OpenAI and the rest of the AI providers have to do next is come up with a secure way of storing all my personal information, using it securely/privately, then convincing everyone it'll be safe for AIs to use. (Believe it or not that last part is the easiest.)
By personal info, I mean my age, gender, sexual preferences, education, health data, home address, relatives and contacts, passwords, bank info and credit cards, SSN, place of work, resume, calendar, email, devices, interests, entertainment and news preferences, etc.
The privacy obsessed HNers will come up with all sorts of reasons this is a really bad idea and impossible to secure from various technical perspectives. It doesn't really matter, because the majority of regular users don't think about it and all that data is out there already. To most people, the loss of privacy/security is worth the benefits, if they think about it at all.