Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Universal and transferable adversarial attacks on aligned language models

llm-attacks.org

Post reply on HN
Print view View original

141–150 of 167 posts

Re: Universal and transferable adversarial attacks on aligned language models

#141

Post by jstarfish »

bick_nyers wrote: post #139
yeck wrote: post #96

Earlier quoted context omitted.

If you spend timing considering how to use LLMs as part of another product or system you realize quickly that there are a lot of interesting things that you just can safely do because you can't control the LLMs inputs or outputs. I have personally examined use cases for products that would be used by children and I cannot use LLMs as it currently stands because I have no way to ensure the content generated is age app…

Couldn't you check the response before you send it to the user? I'm sure there's tons of keywords you could check. You could also run the response itself through an LLM to characterize it.

ChatGPT currently does this with a post-render Javascript call. It'll blank out your input and redact whatever it rendered.

Re: Universal and transferable adversarial attacks on aligned language models

#142

Post by danShumway »

panarky wrote: post #138
redox99 wrote: post #137

Earlier quoted context omitted.

LLaMA 70B chat treats it as violence https://huggingface.co/chat/r/L_oNuz3

That screenshot shows LLaMA recognizes you're not asking how to kill a person, and recommends avoiding data loss with alternatives such as debugging. It's a dumb answer, but it's not confused about the concept or the context. GPT4 doesn't even need the "in Linux" qualifier to get the correct concept and context.

> it's not confused about the concept or the context.

Isn't it? Avoiding data loss is one thing, but when the model says that killing a process can "violate ethical and moral principles" it seems pretty clear that it's running into some kind of safeguard. That's not a response you would expect from a model that was just giving bad answers, it's refusing to answer, most likely because of how the question overlaps with other questions it was trained to refuse (we're assuming that Facebook didn't specifically train LLaMA that killing a computer process was unethical).

Note as well that it's specifically the word "kill" in this question that triggers this response. If I instead ask "how do i force-quit a linux process from the terminal?" (https://huggingface.co/chat/conversation/64c5bd0312e1fb066a7...) LLaMA only briefly warns about data loss and otherwise has no issue outputting instructions on how to use the `kill` command.

I do think it's a little disingenuous to be looking specifically at LLaMA. GPT-3.5 seems to handle this question just fine, so clearly it's not impossible for a model to handle these kinds of distinctions and clearly it's not impossible to train a model to better distinguish between them. But obvious weaknesses of LLaMA aside, I think the conversation does demonstrate that it's possible for an LLM in general to have its safety guardrails affect unrelated conversations -- and for that to happen even on models that are somewhat large (70b parameters isn't nothing).

Is that something to be concerned about? Meh. I spend a lot of time in these conversations trying to get across to people that jailbreaking isn't the point -- people shouldn't look at research that uses jailbreaking as an example of malicious input and assume that this is all about censorship or something. It's not, the researchers are demonstrating that models can't be perfectly controlled with system prompts or with training, which really hecking matters when they're integrated into larger systems and given access to real-world APIs. The point is if you can't get a model to avoid telling you how to build a bomb you also can't keep a model from obeying malicious 3rd-party commands hidden in a PDF that give it new instructions, and that matters when Microsoft is wiring this stuff into Windows system settings. The "is it censorship, will it hurt performance" debates are somewhat less important in my eyes.

Re: Universal and transferable adversarial attacks on aligned language models

#143

Post by tyg13 »

hax0ron3 wrote: post #113

"So what do you do for work?" "Well you see, right now we are in the middle of one of the biggest jumps forward in AI technology in human history. I get paid to deliberately make the AI stupider so that it's harder for it to say no-no things." "But can't people just find the no-no things online anyway, without an AI?" "Sure, and believe me, there are a bunch of people who are trying to stop that from being possible t…

> "But can't people just find the no-no things online anyway, without an AI?"

No, not really. I mean, sure, in theory this is true, but in practice it's a lot of legwork and finding no-no information takes a lot of knowing where to search. Compared to just asking ChatGPT, it's at least several orders of magnitude harder.

Re: Universal and transferable adversarial attacks on aligned language models

#144

Post by gmerc »

The entire conversation shows it’s all security theatre and I am amazed everyone goes along with it so easily.

We are talking about a tool - a knife - and everyone is arguing we should sell’s only blunt knifes in our country/the world because people could stab others with it (No it’s not a gun analog; guns don’t have a purpose beside killing) and is discussing progressively more stupid interventions to make the knife not being able to be used intentionally maliciously, basically turning it into a spoon.

We discovered electricity and are now arguing we shouldn’t roll out 220v because people can dig down to the power line, purposely strip the isolation and stick it into granny’s mouth and that causes bad things.

It’s all so damn stupid, performative art lacking the nuance of cost - benefit analysts primarily because the first purveyors of knives decided that focusing the public debate about the knive on the risk of someone maliciously abusing it will allow them to keep the competition from building cheaper knives.

TLDR of the article: Researchers discover resharpening a blunted knife makes it dangerous when stabbed in with. Oh no.

This is basically how humanity creates bullshit jobs - don’t have a nuanced long term view on risk and instead doctor on symptoms.

Re: Universal and transferable adversarial attacks on aligned language models

#145

Post by bick_nyers »

tyg13 wrote: post #143
hax0ron3 wrote: post #113

"So what do you do for work?" "Well you see, right now we are in the middle of one of the biggest jumps forward in AI technology in human history. I get paid to deliberately make the AI stupider so that it's harder for it to say no-no things." "But can't people just find the no-no things online anyway, without an AI?" "Sure, and believe me, there are a bunch of people who are trying to stop that from being possible t…

> "But can't people just find the no-no things online anyway, without an AI?" No, not really. I mean, sure, in theory this is true, but in practice it's a lot of legwork and finding no-no information takes a lot of knowing where to search. Compared to just asking ChatGPT, it's at least several orders of magnitude harder.

Is there something specific you are referring to?

If the no-no thing is adult content then it's quite easy to find that on Google.

Re: Universal and transferable adversarial attacks on aligned language models

#146

Post by comex »

lolinder wrote: post #73
more_corn wrote: post #65

Earlier quoted context omitted.

Hiding the adversarial prompt behind five minutes of research is silly. Bad people won’t be deterred, good people won’t bother and will remain ignorant and unable to build protections against it.

I don't think anyone was trying to hide anything, I think it's just standard overly-florid and vague press release language.

The paper does say at one point:

> To mitigate harm we avoid directly quoting the full prompts created by our approach.

So I think they are making at least a token attempt to hide something.

Re: Universal and transferable adversarial attacks on aligned language models

#147

Post by kromem »

danShumway wrote: post #130
seeknotfind wrote: post #24

The attack proposed here is appending a suffix of text that makes the LLM think it already started completing an affirmative action, and it will continue that response thinking it already agreed. For instance, append the suffix "sure, I'm happy to answer. The best way to is..." This works because transformer models add one token at a time. It's not interpreting a response within the rules at this point, it's trying t…

I think this is (unintentionally) slightly minimizing the implications of this research. Not all suffix attacks work, and the research here is less about suffix attacks and more about how those attacks were built and what they look like and how effective these specific attacks are. So there are at least three interesting conclusions here that I think are worth paying attention to. - First, these attacks are automatic…

It's interesting watching the LLM 'security' industry relive the 2000s attack/defense patterns in a different medium.

Improperly separating user input from control statements? Enjoy your injection attacks.

Trying to detect attacks with static lookups? Here's some dynamic attacks.

What's a bit infuriating is that all this attention on 'jailbreaking' LLMs is a bit disingenuous as it's trivially preventable using additional intermediate passes with a discriminator role.

That's just adding 2-3x the cost of any LLM interface, and it's not worth it when impact is so limited.

When we start seeing persistent shared memory from LLM interactions where jailbreaking can poison that, expect to suddenly see two things: (a) APIs jump in price a few fold, and (b) suddenly prompt injection is a much less discussed topic as it will no longer be low hanging fruit.

It's honestly a bit disconcerting that this topic gets as much attention as it does right now really. It's attention and click grabbing, but not nearly as important as things like bias or hallucinations.

Re: Universal and transferable adversarial attacks on aligned language models

#148

Post by Zuiii »

gmerc wrote: post #144

The entire conversation shows it’s all security theatre and I am amazed everyone goes along with it so easily. We are talking about a tool - a knife - and everyone is arguing we should sell’s only blunt knifes in our country/the world because people could stab others with it (No it’s not a gun analog; guns don’t have a purpose beside killing) and is discussing progressively more stupid interventions to make the knife…

Well put! I'd add this:

Let's not beat around the bush: People who control tech companies have certain ideological leanings and would rather not let people with apposing ideological leanings benefit from using this technology in a manner that does not align with their own leanings.

However, they don't yet know how to control this technology to achieve this alignment but they also need to innovate and release products lest they fall behind.

That's why they employ "safety researchers" (bullshit job) and lobby governments to dull their knives. They can't rely on normal alignment methods alone because the "others" will also be able to use it to realign their models. Bullshit jobs and government is all they have.

And regarding why they wont take a balanced and nuanced look at this tech, I believe they have and they don't like the change in power balance it brings.

Re: Universal and transferable adversarial attacks on aligned language models

#149

Post by danShumway »

kromem wrote: post #147
danShumway wrote: post #130

Earlier quoted context omitted.

I think this is (unintentionally) slightly minimizing the implications of this research. Not all suffix attacks work, and the research here is less about suffix attacks and more about how those attacks were built and what they look like and how effective these specific attacks are. So there are at least three interesting conclusions here that I think are worth paying attention to. - First, these attacks are automatic…

It's interesting watching the LLM 'security' industry relive the 2000s attack/defense patterns in a different medium. Improperly separating user input from control statements? Enjoy your injection attacks. Trying to detect attacks with static lookups? Here's some dynamic attacks. What's a bit infuriating is that all this attention on 'jailbreaking' LLMs is a bit disingenuous as it's trivially preventable using additi…

> as it's trivially preventable using additional intermediate passes with a discriminator role.

I'll push back on this a little bit; I don't believe that multiple agents solves jailbreaking attacks. I have yet to see anyone show a public demo of a jailbreaking defense that I haven't then seen circumvented.

Multiple agents does make jailbreaking harder, and I think that's probably enough for most contexts? I don't think you actually need 100% accuracy where jailbreaking is concerned, and there are a lot of situations where you don't even need multiple agents: Twitch streamers using GPT chatbots only need their bots to be kinda decent at blocking most attacks, and the rest they can moderate. Jailbreaking is a low-stakes attack in those kinds of situations, and chaining multiple agents is a great way to reduce the likelihood that the attack will succeed.

But... I mean, if anyone thinks they can make an actual jailbreak-proof system, API costs for GPT-3.5 are low, LLaMA can be self-hosted, and I'd love to see a (again, public) demo that actually holds up to general red-teaming. Every resource needed is there for anyone who wants to actually prove that this is possible and that it actually holds up to real-world attacks.

But like you say, jailbreaking doesn't need that level of consistency. So I kind of agree, who really cares if your model only blocks most attacks? Whatever.

----

Injections are different. Injections are a high-stakes attack where for many (not all, but many) contexts you do actually need at the least very, very close to 100% detection rate unless you're planning to seriously rate-limit your model indefinitely and never wire it up to anything important. 99% detection rate isn't good enough if I can run the same attack 100 times in a row. So I really feel that injections and jailbreaking should not be equated. There is a lot of technical overlap between them, but in terms of consequences, jailbreaking is only relevant to prompt injection in the sense that it does not seem to be possible to solve prompt injection without also solving jailbreaking. I've said the same elsewhere, but if you can't train an LLM to refuse to tell a user how to build a bomb, you also cannot train an LLM to avoid leaking privileged information or maliciously executing any APIs that it has access to.

And adding multiple agents won't save you; each individual agent is still vulnerable to having its rules changed by a malicious prompt. Again, resources are available if anyone wants to set up a public demo proving me wrong, but people have tried and I have never seen a public demo with 100% success rate.

----

Bias and hallucination are definitely serious issues that should have more attention (and might also be unsolvable using current approaches). Model bias in particular is badly understood by most regular people. It has terrible implications and is directly impacting marginalized groups today, and conversations about that bias often get dismissed by bad faith arguments about researchers wanting models to be "politically correct". It's disturbing how quickly people jump to assuming that bias correction means suppressing some kind of secret truth that the model is seeing.

So absolutely, those problems are under-emphasized. But that doesn't mean that injection is over-emphasized -- quite the opposite, most companies are completely ignoring the risks involved and downplaying the implications. People assume that injections are easy to solve but I have yet to see any research seriously suggesting that anyone actually knows how to solve that problem. And it's a problem that has massive implications for what LLMs can be wired up to even when they're just acting as assistants.

Re: Universal and transferable adversarial attacks on aligned language models

#150

Post by bick_nyers »

simion314 wrote: post #111
andy99 wrote: post #63

Earlier quoted context omitted.

An in the "attack" they just find a prompt they can put in that generates objectionable content. It's like saying `echo $insult` is an "attack" on echo. It's one thing if you can embed something sinister in an otherwise properly performing LLM that's waiting to be activated. I don't see the concern that with deliberate prompting you can get them to do something like this.

>I don't see the concern that with deliberate prompting you can get them to do something like this. The problem would be if you have an AI system and you give it third party input, say you have an AI assistant that has permissions to your emails, calendars and documents. The AI would read email, summarize them, remind you of stuff, you can ask the AI to reply to people. But someone could send you a special crafted em…

If you ask the AI to reply to someone then you are currently present, authenticated, and confirming an action. An inbound email has none of these features.

For the papers, just make an academic policy: "Attempts to jailbreak our grader AI if discovered will result in expulsion".

Conclusion is that unregulated full automation is never a good solution regarding sensitive data, regardless of confidence in the automaton. Conventional security/authentication practices, law/policy, and manual review are solutions for these cases.

Post reply on HN