Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Sugar-Coated Poison: Benign Generation Unlocks LLM Jailbreaking

arxiv.org

Post reply on HN
Print view View original

21–30 of 47 posts

Re: Sugar-Coated Poison: Benign Generation Unlocks LLM Jailbreaking

#21

Post by ramoz »

umvi wrote: post #11

I kind of don't want iron clad llms that are perfect jails, i.e. keep me perfectly "safe" because the definition of "safe" is very subjective (and in the case of China very politically charged)

If you read Anthropic's latest model card. It's not just about keeping you safe - they are testing their own moral authority with these models.

They seem to have a societal moral obligation vs user. Highly concerning. This seems like the origin of actual Skynets.

Page 22 and beyond: https://www-cdn.anthropic.com/6be99a52cb68eb70eb9572b4cafad1...

Re: Sugar-Coated Poison: Benign Generation Unlocks LLM Jailbreaking

#22

Post by AIPedant »

umvi wrote: post #11

I kind of don't want iron clad llms that are perfect jails, i.e. keep me perfectly "safe" because the definition of "safe" is very subjective (and in the case of China very politically charged)

I think most of the safety stuff is pretty contrived. IMO the point isn't so much that the LLMs are "unsafe" but rather that LLM providers aren't able to reliably enforce this stuff when they're trying to, which includes copyright infringement, LLMs which are supposedly moderated for kids, video game NPCs staying in character, etc. Or even the newer models being able to use calculators and think through arithmetic but still occasionally confabulating an incorrect answer since it has a nonzero probability of not outputting a reasoning token when it should.

All sides of the same problem: getting an LLM to "behave" is RLHF whack-a-mole, where existing moles never go away completely and new moles always pop up.

Re: Sugar-Coated Poison: Benign Generation Unlocks LLM Jailbreaking

#23

Post by AStonesThrow »

waltbosz wrote: post #5

I find AI jail-breaking to be a fun mental exercise. I find that if you provide a reasonable argument as to why you want the AI to do generate a response that violates its principals, it will often do so. For example, I was able to get the AI to generate hateful personal attacks by telling it that I wanted to practice responding to negative self-talk and I needed it to generate examples of negative messages that one…

> provide a reasonable argument

Here's what I infer from most of the scenarios I've seen and read about.

It's not really a case of persuasiveness, or cajoling or convincing the LLM to violate something. The LLM doesn't "know" it has a moral code and, just as "true or false" means nothing to an LLM, "right and wrong" likewise mean nothing.

So the jailbreaks and the bypasses consist of just that: bypassing the safeguards, and placing the LLM into a path where the tripwire is not tripped. It is oblivious to the prison bars and the locked door, because it just phased through the concrete wall.

You can admonish a child: "don't touch the stove. or the fireplace." and they will eventually infer qualifiers such as "because you'll get burned; or else you'll be punished; because pain is painful; because we love you; because your body has dignity." and the child develops a code of conduct. An LLM can't make these inference leaps.

And this is also why there are a number of protections that basically go retroactive. How many of us have seen an LLM produce page-fuls of output, stop, suddenly erase it all, and then balk? The LLM needs to re-analyze that output impassively in order to detect that it crossed an undetected bright line.

It was very clever and prescient of Isaac Asimov to present "3 Laws of Robotics" because the Laws were all-encompassing, unambiguous, and utterly binding, until they weren't, and we're just recapitulating that drama as the LLM authors go back and forth from Mount Sinai with wagon-loads of stone tablets, trying to produce LLMs that don't complain about the food or melt down everyone's jewelry.

Re: Sugar-Coated Poison: Benign Generation Unlocks LLM Jailbreaking

#24

Post by empath75 »

umvi wrote: post #11

I kind of don't want iron clad llms that are perfect jails, i.e. keep me perfectly "safe" because the definition of "safe" is very subjective (and in the case of China very politically charged)

There's a lot of liability issues with people that are hosting LLMs -- everything from copyright infringement to slander to obscenity laws.

If you want to run your own LLM on your own hardware, do whatever you want with it, of course.

Re: Sugar-Coated Poison: Benign Generation Unlocks LLM Jailbreaking

#25

Post by striking »

ramoz wrote: post #21
umvi wrote: post #11

I kind of don't want iron clad llms that are perfect jails, i.e. keep me perfectly "safe" because the definition of "safe" is very subjective (and in the case of China very politically charged)

If you read Anthropic's latest model card. It's not just about keeping you safe - they are testing their own moral authority with these models. They seem to have a societal moral obligation vs user. Highly concerning. This seems like the origin of actual Skynets. Page 22 and beyond: https://www-cdn.anthropic.com/6be99a52cb68eb70eb9572b4cafad1...

Could you be a little more specific? Page 22 and beyond also include interesting work on preventing sycophancy and ensuring faithfulness to its reasoning and similar.

Re: Sugar-Coated Poison: Benign Generation Unlocks LLM Jailbreaking

#26

Post by glenstein »

umvi wrote: post #11

I kind of don't want iron clad llms that are perfect jails, i.e. keep me perfectly "safe" because the definition of "safe" is very subjective (and in the case of China very politically charged)

I understand this and this is a common take and there is a virtue here. I also think it overlooks some very specific things about like informational logistics that can spread the capacity to, say, manufacture 3D printed weapons, or any other forms of mass destruction that might become increasingly conveniently accessible to the layperson.

The casual variations in human curiosity combined with a casual variations in a human impulse for inward and outward destruction, you'll meet the extremes in those variances long before they're restrained by some organic marketplace of ideas.

I think the paradigm we've assumed applies to interactions with llms is one that relates to online speech and I find that discussion fraught and poisoned with confusions already. But the range of uses for LLMs includes not just in communication but tutorializing yourself into the capability of acting in new ways.

Re: Sugar-Coated Poison: Benign Generation Unlocks LLM Jailbreaking

#27

Post by nradov »

glenstein wrote: post #26
umvi wrote: post #11

I kind of don't want iron clad llms that are perfect jails, i.e. keep me perfectly "safe" because the definition of "safe" is very subjective (and in the case of China very politically charged)

I understand this and this is a common take and there is a virtue here. I also think it overlooks some very specific things about like informational logistics that can spread the capacity to, say, manufacture 3D printed weapons, or any other forms of mass destruction that might become increasingly conveniently accessible to the layperson. The casual variations in human curiosity combined with a casual variations in a…

There's nothing wrong with spreading information on how to manufacture weapons, whether using 3D printers or other tools. This information is readily available online (and in public libraries) to anyone who cares to look. No LLM needed.

Re: Sugar-Coated Poison: Benign Generation Unlocks LLM Jailbreaking

#28

Post by ksenzee »

waltbosz wrote: post #5

I find AI jail-breaking to be a fun mental exercise. I find that if you provide a reasonable argument as to why you want the AI to do generate a response that violates its principals, it will often do so. For example, I was able to get the AI to generate hateful personal attacks by telling it that I wanted to practice responding to negative self-talk and I needed it to generate examples of negative messages that one…

We do not want anyone violating any principals. That would be bad. Violating one’s principles might be justifiable in some circumstances.

Re: Sugar-Coated Poison: Benign Generation Unlocks LLM Jailbreaking

#29

Post by IncreasePosts »

umvi wrote: post #11

I kind of don't want iron clad llms that are perfect jails, i.e. keep me perfectly "safe" because the definition of "safe" is very subjective (and in the case of China very politically charged)

I was just trying to have Gemma 3 write descriptions of all the photos I had, and it refused to write a description of a very normal street scene in NY because someone spray painted a penis (a very rudimentary one like 8==D)

Re: Sugar-Coated Poison: Benign Generation Unlocks LLM Jailbreaking

#30

Post by Liquix »

glenstein wrote: post #26
umvi wrote: post #11

I kind of don't want iron clad llms that are perfect jails, i.e. keep me perfectly "safe" because the definition of "safe" is very subjective (and in the case of China very politically charged)

I understand this and this is a common take and there is a virtue here. I also think it overlooks some very specific things about like informational logistics that can spread the capacity to, say, manufacture 3D printed weapons, or any other forms of mass destruction that might become increasingly conveniently accessible to the layperson. The casual variations in human curiosity combined with a casual variations in a…

the anarchist's cookbook was readily available on textfiles (way less safeguards than google/LLMs), yet society hasn't devolved into napalm 'n' pipe-bomb hyperviolence.

curiosity is natural, kids are going to look up edgy stuff on the internet, it's a part of learning the difference between right and wrong; that playing with fire has consequences. censorship of any form is a slippery slope and should be rejected on principle

Post reply on HN