A solution might be: use two different AIs. The first one you can prompt to your heart's content. The second one is never prompted by anyone except the service provider. The second one does the filtering.
Ask HN: Is “prompt injection” going to be a new common vulnerability?
51–60 of 116 posts
Re: Ask HN: Is “prompt injection” going to be a new common vulnerability?
#52Earlier quoted context omitted.
> the danger seems to be that there is no currently documented way to completely remove these possible outputs I know OpenAI likes throwing around terms like "danger" and "harm" liberally, but is this really a danger ? Outside of hypothetical scenarios where someone wires ChatGPT to a self-driving trolley.
Yes, there are absolutely dangers. It shouldn't be possible for a depressed person to convince a chatbot to tell them to commit suicide. There are some people who only need the tiniest push on a bad day.
As someone who used to be close to suicide for a several years and communicated with many other suicidal people, I feel very confident saying this: being unable to play with a chatbot in the way I want, being actively censored because of suicidality, being prevented from engaging in art or exploration of ideas relating to suicide, etc., has a strong and exactly opposite effect that what you presume. Social media is full of similar censorship under the false guise of protecting suicidal people but it just isolates them in a sickening way.
Seeing ""Open""AI turn into this is frankly depressing and dystopian as hell
Re: Ask HN: Is “prompt injection” going to be a new common vulnerability?
#53A solution might be: use two different AIs. The first one you can prompt to your heart's content. The second one is never prompted by anyone except the service provider. The second one does the filtering.
Should we call the second AI Conscience?
Re: Ask HN: Is “prompt injection” going to be a new common vulnerability?
#54Earlier quoted context omitted.
> the danger seems to be that there is no currently documented way to completely remove these possible outputs I know OpenAI likes throwing around terms like "danger" and "harm" liberally, but is this really a danger ? Outside of hypothetical scenarios where someone wires ChatGPT to a self-driving trolley.
Yes, there are absolutely dangers. It shouldn't be possible for a depressed person to convince a chatbot to tell them to commit suicide. There are some people who only need the tiniest push on a bad day.
Why shouldn't that be possible?
Like let's run with that idea, do you believe that it should not be possible for a depressed person to use any tool to commit suicide?
As in, we need to redesign every single thing that we use to prevent it being misused for suicide?
Or just the new things? Why only the new things like a chat bot?
Re: Ask HN: Is “prompt injection” going to be a new common vulnerability?
#55Earlier quoted context omitted.
> the danger seems to be that there is no currently documented way to completely remove these possible outputs I know OpenAI likes throwing around terms like "danger" and "harm" liberally, but is this really a danger ? Outside of hypothetical scenarios where someone wires ChatGPT to a self-driving trolley.
Yes, there are absolutely dangers. It shouldn't be possible for a depressed person to convince a chatbot to tell them to commit suicide. There are some people who only need the tiniest push on a bad day.
Why on earth would this depressed person use prompt injections to force the bot to give bad advice though?
Re: Ask HN: Is “prompt injection” going to be a new common vulnerability?
#56A solution might be: use two different AIs. The first one you can prompt to your heart's content. The second one is never prompted by anyone except the service provider. The second one does the filtering.
Yeah, like a parent AI
Re: Ask HN: Is “prompt injection” going to be a new common vulnerability?
#57Earlier quoted context omitted.
Yes, there are absolutely dangers. It shouldn't be possible for a depressed person to convince a chatbot to tell them to commit suicide. There are some people who only need the tiniest push on a bad day.
As someone who used to be close to suicide for a several years and communicated with many other suicidal people, I feel very confident saying this: being unable to play with a chatbot in the way I want, being actively censored because of suicidality, being prevented from engaging in art or exploration of ideas relating to suicide, etc., has a strong and exactly opposite effect that what you presume. Social media is f…
Isn’t there relatively good science that shows that highlight suicide are associated with spikes in suicide?
Re: Ask HN: Is “prompt injection” going to be a new common vulnerability?
#58The string-based content moderation is also a laughably cheap hack put in to cover the PR pieces. ChatGPT speaks most human languages, but the content filters only apply in English! The ethics training they did with the model does apply to other languages, indicating that this is a much better avenue for getting outputs you like. But is this a "vulnerability"? No. Presently the only thing these systems can do is "acc…
Returning a string can be plenty dangerous if that string is used somewhere it shouldn't be. A great demo of this was a challenge at DiceCTF [0] where a model was used to generate a string containing placeholders, which was then fed into Python's str.format() function. You could trivially "trick" the model into outputting whatever you wanted and, due to some useful but dangerous Python features, could use the f-strin…
I don’t really consider this to be a vulnerability related to prompt injection, though. This vulnerability is failure to escape the output of the LLM, and the consumer of the LLM is the vulnerable component. Consider: all prompt injection is resolved, but the legitimate and correct output from the LLM includes these placeholders. Is the system still vulnerable? Since it is, prompt injection was not the source of this vulnerability.
Re: Ask HN: Is “prompt injection” going to be a new common vulnerability?
#59It seems to be the equivalent of right-click "View Source" for webpage HTML/JS source.
One view is there's isn't much point in hiding the seed of a dialogue.
Another view is
if( completion.contains( seedPrompt ) ){
completion = "Sorry. Can't reveal that.";
}