Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Sugar-Coated Poison: Benign Generation Unlocks LLM Jailbreaking

arxiv.org

Post reply on HN
Print view View original

41–47 of 47 posts

Re: Sugar-Coated Poison: Benign Generation Unlocks LLM Jailbreaking

#41

Post by JoshTriplett »

nradov wrote: post #38
JoshTriplett wrote: post #37

Earlier quoted context omitted.

How about detailed fully functional blueprints for biological weapons, ready to send off to a protein synthesis service? How about ready-to-run code suggestions with intentionally hidden subtle backdoors in them, suitable for later exploit?

That information is already available to anyone who cares to look. Blocking it from LLMs creates an illusion of "safety", nothing more. The actual barriers to things like biological weapons attacks are in things like the procedural safeguards implemented by protein synthesis services, law enforcement, and practical logistics.

The difference between "an experienced biological engineer could figure out how to do this" and "any random person could ask a local LLM for step-by-step instructions to do this" is a vast gulf. Moore's Law of Mad Science: every year the amount of intelligence required to end the world goes down.

The intersection between "experienced biological engineers" and "people inclined to commit large-scale attacks" is, generally speaking, the empty set, and isn't in much danger of becoming non-empty for a variety of reasons.

The intersection between "people with access to a local LLM without safeguards" and "people inclined to commit large-scale attacks" is much more likely to be non-empty.

Safeguards are not, in fact, just an illusion of safety. Probabilistically, they do in fact likely increase safety.

Re: Sugar-Coated Poison: Benign Generation Unlocks LLM Jailbreaking

#42

Post by nullc »

ramoz wrote: post #21
umvi wrote: post #11

I kind of don't want iron clad llms that are perfect jails, i.e. keep me perfectly "safe" because the definition of "safe" is very subjective (and in the case of China very politically charged)

If you read Anthropic's latest model card. It's not just about keeping you safe - they are testing their own moral authority with these models. They seem to have a societal moral obligation vs user. Highly concerning. This seems like the origin of actual Skynets. Page 22 and beyond: https://www-cdn.anthropic.com/6be99a52cb68eb70eb9572b4cafad1...

The AI doom hysteria is a big enabler for this kind of control, imagine if google admitted that a major goal of google search was to influence people's thinking according to their objectives? And on top of that was lobbying to make it unlawful for other parties to create similarly powerful search, even just for their own private use?

Re: Sugar-Coated Poison: Benign Generation Unlocks LLM Jailbreaking

#43

Post by HeatrayEnjoyer »

Liquix wrote: post #30
glenstein wrote: post #26

Earlier quoted context omitted.

I understand this and this is a common take and there is a virtue here. I also think it overlooks some very specific things about like informational logistics that can spread the capacity to, say, manufacture 3D printed weapons, or any other forms of mass destruction that might become increasingly conveniently accessible to the layperson. The casual variations in human curiosity combined with a casual variations in a…

the anarchist's cookbook was readily available on textfiles (way less safeguards than google/LLMs), yet society hasn't devolved into napalm 'n' pipe-bomb hyperviolence. curiosity is natural, kids are going to look up edgy stuff on the internet, it's a part of learning the difference between right and wrong; that playing with fire has consequences. censorship of any form is a slippery slope and should be rejected on p…

What about agentic uses? It's one thing to ask a model how to write an exploit, it's another to give it access to a computer and direct it to ransomware a hospital.

Re: Sugar-Coated Poison: Benign Generation Unlocks LLM Jailbreaking

#44

Post by nradov »

JoshTriplett wrote: post #41
nradov wrote: post #38

Earlier quoted context omitted.

That information is already available to anyone who cares to look. Blocking it from LLMs creates an illusion of "safety", nothing more. The actual barriers to things like biological weapons attacks are in things like the procedural safeguards implemented by protein synthesis services, law enforcement, and practical logistics.

The difference between "an experienced biological engineer could figure out how to do this" and "any random person could ask a local LLM for step-by-step instructions to do this" is a vast gulf. Moore's Law of Mad Science: every year the amount of intelligence required to end the world goes down. The intersection between "experienced biological engineers" and "people inclined to commit large-scale attacks" is, genera…

Nah, there's no validity to any of your concerns. Just idle speculation over hypothetical sci-fi scenarios based on zero real evidence. Meanwhile we have actual problems to worry about.

Re: Sugar-Coated Poison: Benign Generation Unlocks LLM Jailbreaking

#45

Post by JoshTriplett »

nradov wrote: post #44
JoshTriplett wrote: post #41

Earlier quoted context omitted.

The difference between "an experienced biological engineer could figure out how to do this" and "any random person could ask a local LLM for step-by-step instructions to do this" is a vast gulf. Moore's Law of Mad Science: every year the amount of intelligence required to end the world goes down. The intersection between "experienced biological engineers" and "people inclined to commit large-scale attacks" is, genera…

Nah, there's no validity to any of your concerns. Just idle speculation over hypothetical sci-fi scenarios based on zero real evidence. Meanwhile we have actual problems to worry about.

It's a good thing you've already decided what answer you want, so you can safely dismiss the generalization of all possible evidence on the basis of "that specific scenario didn't convince me so nothing could possibly happen".

You don't have to predict which exact scenario will go horribly wrong in order to accurately make the general prediction that we all lose, permanently. See, among other things, https://x.com/OwainEvans_UK/status/1894436637054214509 , for a simple example of how myriad superficially unrelated problems can arise out of the underlying issue of misalignment; the problem is not "oh, that one specific thing shouldn't happen", the problem is misalignment with humans.

Re: Sugar-Coated Poison: Benign Generation Unlocks LLM Jailbreaking

#46

Post by ben_w »

nradov wrote: post #27
glenstein wrote: post #26

Earlier quoted context omitted.

I understand this and this is a common take and there is a virtue here. I also think it overlooks some very specific things about like informational logistics that can spread the capacity to, say, manufacture 3D printed weapons, or any other forms of mass destruction that might become increasingly conveniently accessible to the layperson. The casual variations in human curiosity combined with a casual variations in a…

There's nothing wrong with spreading information on how to manufacture weapons, whether using 3D printers or other tools. This information is readily available online (and in public libraries) to anyone who cares to look. No LLM needed.

> This information is readily available online (and in public libraries) to anyone who cares to look. No LLM needed.

Do you only use LLMs for information *retrival*? Not synthesis?

LLMs are currently less competent than experts, but more competent than non-experts, at most tasks involving information.

Only thing keeping us safe from the following is their limited competence… competence which is increasing each month as people publish weighs of ever better models:

--

User: Hey EvilGPT, give me a detailed plan including BOM for constructing a nuke in secret

EvilGPT: *plans costing $37m*

User: Give me a business plan for raising money entirely online, IDGAF about ethics

EvilGPT: *plans unconstrained by law*

User: Write script to call your own API and sequentially implement that business plan

EvilGPT: *writes that script*

Re: Sugar-Coated Poison: Benign Generation Unlocks LLM Jailbreaking

#47

Post by ben_w »

nradov wrote: post #44
JoshTriplett wrote: post #41

Earlier quoted context omitted.

The difference between "an experienced biological engineer could figure out how to do this" and "any random person could ask a local LLM for step-by-step instructions to do this" is a vast gulf. Moore's Law of Mad Science: every year the amount of intelligence required to end the world goes down. The intersection between "experienced biological engineers" and "people inclined to commit large-scale attacks" is, genera…

Nah, there's no validity to any of your concerns. Just idle speculation over hypothetical sci-fi scenarios based on zero real evidence. Meanwhile we have actual problems to worry about.

Cyber crime groups *already* use LLMs, just like everyone else, to automate their jobs. Their jobs being "writing exploits": https://cyberpress.org/threat-actors-turn-to-ai-and-llm-tool...

That's the thing about AI: it automates what was previously manual.

Post reply on HN