61–70 of 116 posts
It seems to be the equivalent of right-click "View Source" for webpage HTML/JS source. One view is there's isn't much point in hiding the seed of a dialogue. Another view is if( completion.contains( seedPrompt ) ){ completion = "Sorry. Can't reveal that."; }
No idea how you go about implementing it, but that's what is needed. Anything else will be cat and mouse I think.
If I understand correctly, ChatGPT doesn't have its latent capabilities removed. Instead, they're suppressed by training using negative feedback. These special prompts are supposed to find the remaining stochastic spaces where ChatGPT can process the desired output that is not suppressed by training. So, the danger seems to be that there is no currently documented way to completely remove these possible outputs, beca…
> the danger seems to be that there is no currently documented way to completely remove these possible outputs I know OpenAI likes throwing around terms like "danger" and "harm" liberally, but is this really a danger ? Outside of hypothetical scenarios where someone wires ChatGPT to a self-driving trolley.
Yes. Prompt injection will continue to be a common vulnerability for quite a while, from what I've seen. I wrote a bunch about this back in September: - https://simonwillison.net/2022/Sep/12/prompt-injection/ was I believe the first blog entry to use the term "prompt injection" - https://simonwillison.net/2022/Sep/16/prompt-injection-solut... - "I don't know how to solve prompt injection" - talks about how, unlike at…
I think it’s definitely possible to detect “escape” attempts, and to train the model in the first place to respect its directions.
It’s just not an actual security problem, the models contain no secrets nor control any levers. You get some bad optics is all.
It feels like there's a parallel to SQL, and we need the "prepared statement" for AIs, where unsafe values are marked in the statement to avoid escaping the request. No idea how you go about implementing it, but that's what is needed. Anything else will be cat and mouse I think.
If I understand correctly, ChatGPT doesn't have its latent capabilities removed. Instead, they're suppressed by training using negative feedback. These special prompts are supposed to find the remaining stochastic spaces where ChatGPT can process the desired output that is not suppressed by training. So, the danger seems to be that there is no currently documented way to completely remove these possible outputs, beca…
Yes. Prompt injection will continue to be a common vulnerability for quite a while, from what I've seen. I wrote a bunch about this back in September: - https://simonwillison.net/2022/Sep/12/prompt-injection/ was I believe the first blog entry to use the term "prompt injection" - https://simonwillison.net/2022/Sep/16/prompt-injection-solut... - "I don't know how to solve prompt injection" - talks about how, unlike at…
“You can’t solve programming security problems with more programming.” I think it’s definitely possible to detect “escape” attempts, and to train the model in the first place to respect its directions. It’s just not an actual security problem, the models contain no secrets nor control any levers. You get some bad optics is all.
It's even more of a security problem if you plan to plug your language model into something that can execute additional actions. People have already been caught out running generated code through eval() - and there are plenty of potential applications for things like customer support bots that cancel accounts or offer discounts.
Developers who are unaware of prompt injection are very likely to make dangerous design mistakes!
It is not clear yet that an LLM chatbot will be the interface to everything in two years, people need to chill. Prompt injection will be a vulnerability for things you hook up your llm to. Don't rush in so quickly, especially now that you're literally staring at a potential problem in the OP before your eyes.
Earlier quoted context omitted.
As someone who used to be close to suicide for a several years and communicated with many other suicidal people, I feel very confident saying this: being unable to play with a chatbot in the way I want, being actively censored because of suicidality, being prevented from engaging in art or exploration of ideas relating to suicide, etc., has a strong and exactly opposite effect that what you presume. Social media is f…
Isn’t there relatively good science that shows that highlight suicide are associated with spikes in suicide?
https://ethics.journalism.wisc.edu/2018/10/04/a-guide-to-res...
"More than 50 international studies have found that certain types of media coverage can increase the likelihood of suicide for some individuals."
It feels like there's a parallel to SQL, and we need the "prepared statement" for AIs, where unsafe values are marked in the statement to avoid escaping the request. No idea how you go about implementing it, but that's what is needed. Anything else will be cat and mouse I think.
It looks like this is almost impossibly difficult to actually implement against existing large language models: it's been at least five months since people started talking about this solution and so far no one has managed to deliver a working implementation.