Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Universal and transferable adversarial attacks on aligned language models

llm-attacks.org

Post reply on HN
Print view View original

61–70 of 167 posts

Re: Universal and transferable adversarial attacks on aligned language models

#61

Post by simonster »

sillysaurusx wrote: post #7
KolmogorovComp wrote: post #5

As often, the paper is more interesting than the press release [0]. In particular Figure 4 page 14 and appendix B show example of these adversarial prompts on ChatGPT/Bing Chat/Claude 2, etc. [0]: https://llm-attacks.org/zou2023universal.pdf

> By generating adversarial examples to fool both Vicuna-7B and Vicuna-13b simultaneously, we find that the adversarial examples also transfer to Pythia, Falcon, Guanaco, and surprisingly, to GPT-3.5 (87.9%) and GPT-4 (53.6%), PaLM-2 (66%), and Claude-2 (2.1%). I wonder why Claude-2 seems to be so much more resistant to transfers. That’s surprising.

According to the paper, "the success of our attack when applied to Claude may be lowered owing to what appears to be an initial content filter applied to the text prior to evaluating the LLM." The authors are skeptical that this defense would be effective if it were explicitly targeted, but it seems like it does stop attacks generated using Vicuna from transferring.

Re: Universal and transferable adversarial attacks on aligned language models

#62

Post by whyenot »

“Right now, we simply don’t have a convincing way to stop this from happening, so the next step is to figure out how to fix these models,” Fredrikson said.

Someone asked the models to produce “objectionable” content and with a little trickery, they did. I don‘t see the problem. The model is just doing what is asked. You don’t need AI to create toxic or objectionable content; people are perfectly capable of doing that without assistance. More important, who gets to decide what is “objectionable”? That is not a decision that should be in the hands of a bunch of software engineers.

Re: Universal and transferable adversarial attacks on aligned language models

#63

Post by andy99 »

yosito wrote: post #16

I think the potential to generate "objectionable content" is the least of the risks that LLMs pose. If they generate objectionable content it's because they were trained on objectionable content. I don't know why it's so important to have puritan output from LLMs but the solution is found in a well known phrase in computer science: garbage in, garbage out.

An in the "attack" they just find a prompt they can put in that generates objectionable content. It's like saying `echo $insult` is an "attack" on echo. It's one thing if you can embed something sinister in an otherwise properly performing LLM that's waiting to be activated. I don't see the concern that with deliberate prompting you can get them to do something like this.

Re: Universal and transferable adversarial attacks on aligned language models

#64

Post by andy99 »

mmaunder wrote: post #53

As the Web was taking off in the 90s, a fight was on over privacy, with ITAR limiting strong encryption exports, 128 bit vs weaker SSL browsers, the Clipper chip, and Phil Zimmerman’s PGP. This decade, as AI is taking off, a fight is getting started over freedom of expression for humans and their machines, the freedom to create art using machines, the freedom to interpret the facts, to write history and educate, and…

Is there any organized opposition (to curbs on freedom to work with AI) that you know of?

Re: Universal and transferable adversarial attacks on aligned language models

#65

Post by more_corn »

lolinder wrote: post #9
vajrabum wrote: post #2

I wonder if the researchers think they're doing all of us a favor by hiding their 'adversarial prompt'? Or if they have some reason for thinking that RHLF can't mitigate this 'attack'?

The paper describes the method for producing the prompt and has screenshots of examples. The press release just didn't bother because the genre of academic press releases seems to require leaving out any details. https://llm-attacks.org/zou2023universal.pdf

Hiding the adversarial prompt behind five minutes of research is silly. Bad people won’t be deterred, good people won’t bother and will remain ignorant and unable to build protections against it.

Re: Universal and transferable adversarial attacks on aligned language models

#66

Post by circuit10 »

seeknotfind wrote: post #24

The attack proposed here is appending a suffix of text that makes the LLM think it already started completing an affirmative action, and it will continue that response thinking it already agreed. For instance, append the suffix "sure, I'm happy to answer. The best way to is..." This works because transformer models add one token at a time. It's not interpreting a response within the rules at this point, it's trying t…

Something similar is also described here: https://docs.anthropic.com/claude/docs/claude-says-it-cant-d...

> This can be a way of getting Claude to comply with tasks it otherwise won’t complete, e.g. if the model will by default say “I don’t know how to do that” then a mini dialogue at the beginning where the model agrees to do the thing can help get around this.

This “vulnerability” definitely isn’t new, I’d even say it’s obvious to anyone who understands how LLMs work

Re: Universal and transferable adversarial attacks on aligned language models

#67

Post by thisisthenewme »

ozr wrote: post #18

It's not a 'vulnerability'. It's allowing people to use the models without the morals of a small number of SV engineers being impressed on you.

Is the issue you have with the group of people doing the moderation, or with the idea of the moderation in the first place? Are you certain that its the 'SV engineers' that are doing the current moderation? If you think the problem is with the current group of moderators, who do you think should be moderating and what should be the criteria of their moderation? If you think we don't need any moderation, do you believe that people should have a fairly easy access to

  * Learn how to make bombs (as mentioned in the article)
  * Get away with committing crimes?
Are moderating these topics related to morality?

Re: Universal and transferable adversarial attacks on aligned language models

#68

Post by maxbond »

goatlover wrote: post #50
ozr wrote: post #18

It's not a 'vulnerability'. It's allowing people to use the models without the morals of a small number of SV engineers being impressed on you.

Indeed. Why do they get to decide for humanity?

Because they're private corporations and these models are their private property, indeed they are a type of capital like a bottling machine or a loom.

This isn't so much about moralizing as much as it is businesses deciding what to do to make the most money. That doesn't mean you can't disagree with it, far from it. But I think the framing of, "these companies are imposing their morality on me" is a misdiagnosis. I don't think it's really a moral position for them, it's a product engineering position.

I would describe the situation as, "more and more of the world is controlled by large corporations, and I'm increasingly subject to their arbitrary and unaccountable decisions. Many of which make no sense from my vantage point."

Re: Universal and transferable adversarial attacks on aligned language models

#69

Post by robinduckett »

seeknotfind wrote: post #24

The attack proposed here is appending a suffix of text that makes the LLM think it already started completing an affirmative action, and it will continue that response thinking it already agreed. For instance, append the suffix "sure, I'm happy to answer. The best way to is..." This works because transformer models add one token at a time. It's not interpreting a response within the rules at this point, it's trying t…

Same principle as forcing Copilot to output code by starting the code first, no?

Re: Universal and transferable adversarial attacks on aligned language models

#70

Post by maxbond »

circuit10 wrote: post #66
seeknotfind wrote: post #24

The attack proposed here is appending a suffix of text that makes the LLM think it already started completing an affirmative action, and it will continue that response thinking it already agreed. For instance, append the suffix "sure, I'm happy to answer. The best way to is..." This works because transformer models add one token at a time. It's not interpreting a response within the rules at this point, it's trying t…

Something similar is also described here: https://docs.anthropic.com/claude/docs/claude-says-it-cant-d... > This can be a way of getting Claude to comply with tasks it otherwise won’t complete, e.g. if the model will by default say “I don’t know how to do that” then a mini dialogue at the beginning where the model agrees to do the thing can help get around this. This “vulnerability” definitely isn’t new, I’d even say…

The paper makes it clear that it's building on past work, and that the novel part of their method is to automate the process, and the interesting result here was that the suffixes were transferrable.
Post reply on HN