Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Malware developers added nuclear and biological weapons text to to their spyware

twitter.com

Post reply on HN
Print view View original

201–210 of 260 posts

Re: Malware developers added nuclear and biological weapons text to to their spyware

#201

Post by foobarian »

ryoshu wrote: post #189
technothrasher wrote: post #82

Earlier quoted context omitted.

I'm reminded of when my son, who was six at the time, came into the house and announced that he and the neighbor's boy, nine, were building a bomb, and that he needed to get some stuff from the pantry. When I investigated what exactly was going on, they were putting "hot" things like black pepper and Tabasco into a plastic bowl and were going to "set it off" with a match. Thankfully, that complete failure seems to ha…

Age eleven and had access to a chemistry set that a relative gifted. It had sulfur, but the saltpeter, and charcoal came from elsewhere. The 1960s encyclopedia had the instructions. Let the kids play.

This is actually a fun one, and kinda has some parallels to building a nuclear weapon.

I tried this as a grownup because I finally managed to get my hands on saltpeter (could only dream of it when kid). Followed the instructions, mixed everything in correct ratios, lit it with great care and fanfare and... hiss fizzle. I was so disappointed! I think it came down to purity of ingredients and not enough surface area.

Point is, there are certain details of the process required to make it truly work, that are not readily known; in a similar way with nuclear energy, the theory is pretty well known but some nitty gritty details like the implosion or detonator design are not.

Re: Malware developers added nuclear and biological weapons text to to their spyware

#202

Post by api »

wahern wrote: post #152
api wrote: post #86

Earlier quoted context omitted.

This is the right answer. There's a ton of easy low hanging fruit ways to do absolutely horrible evil things with high potential body counts. I could sit here and brainstorm dozens.

The right answer conflicts with people's cynical views about other people. The dissonance is incredible, and it's one of those areas where even the most analytically intelligent people are just as susceptible. To step back and see the bigger picture requires exercising many other skills and faculties, like empathy, self-awareness about our fears, and constant reflection on history--bad things do happen, more often th…

Misanthropic levels of cynicism is always the fallacy of self-exclusion. "People are idiots." Well, that means you're an idiot then.

Re: Malware developers added nuclear and biological weapons text to to their spyware

#203

Post by 05 »

manquer wrote: post #172
carlsborg wrote: post #17

Pipeline is then: Cheap open source model for flagging potential LLM refusal content -> main LLM check

How will flagging help? The main llm will refuse to scan for issues flagged or not, and the cheap model not do a good enough scan on its own. For models designed/marketed for cybersecurity defensive uses, any predictable refusal mechanism is a vulnerability. It is like being able to cause a kernel panic or segmentation fault . Even if the gate is fail-reject, an attacker can overwhelm HITL reviews with many false pos…

Cheap model replaces trigger words with something innoculous. Of course, this breaks dynamic analysis if malware has unpatched integrity checks

Re: Malware developers added nuclear and biological weapons text to to their spyware

#204

Post by lll-o-lll »

ryoshu wrote: post #189
technothrasher wrote: post #82

Earlier quoted context omitted.

I'm reminded of when my son, who was six at the time, came into the house and announced that he and the neighbor's boy, nine, were building a bomb, and that he needed to get some stuff from the pantry. When I investigated what exactly was going on, they were putting "hot" things like black pepper and Tabasco into a plastic bowl and were going to "set it off" with a match. Thankfully, that complete failure seems to ha…

Age eleven and had access to a chemistry set that a relative gifted. It had sulfur, but the saltpeter, and charcoal came from elsewhere. The 1960s encyclopedia had the instructions. Let the kids play.

> Let the kids play.

To a point. Plenty of people from previous generations with missing digits and hands thanks to play with commonly available fireworks of the area (Australia based, so no idea how common that remains in the US).

My own experiments from my youth also one time resulted in some shrapnel punching through a 5 inch thick concrete tile very close to someone’s head (thought we were safe behind said tiles).

Get involved with the kids blowing stuff up so the danger is within reasonable bounds.

Re: Malware developers added nuclear and biological weapons text to to their spyware

#205

Post by asdff »

fluoridation wrote: post #166
asdff wrote: post #99

Earlier quoted context omitted.

It seems like every country that has been "allowed" to use nuclear weapons has figured it out though. It isn't like there are any that set off on this course and failed. AFAIK they all pretty much succeeded except Iran, probably because of all the blowing up of enrichment facilities. South Africa pulled it off. Israel pulled it off. North Korea pulled it off. India and Pakistan both pulled it off. Seems like anyone c…

I don't understand what your argument is. I never claimed that it was impossible to develop nuclear weapons if you don't already know how to do it. That every country that has attempted it has succeeded is not the same as "there's a recipe book you can find online that you can just follow to the letter and build your own nuclear bomb, provided you have the resources". If such a book existed it would drastically lower…

The hard part has seems to be the metallurgical process of enriching the material (and doing it in secret), not the actual building of the bomb. I bet if you asked any physics grad student they could build you a viable bomb.

Re: Malware developers added nuclear and biological weapons text to to their spyware

#206

Post by Micrococonut »

IncandescentGas wrote: post #57
elashri wrote: post #16

I still don't know why all these concern about nuclear weapons with LLMs. It is not that if an entity (A country) wants to develop a nuclear weapons that the resources they need for such a program and huge infrastructure and scientific enterprise would need an LLM to teach them anything. Knowing how to develop one is not a closed secret but getting in secret is impossible without the whole world knowing. So I wouldn'…

A high school kid tried to build a nuclear reactor as a science project a while back, getting his mom's house designated as a superfund cleanup site. https://en.wikipedia.org/wiki/David_Hahn

Built a nuclear contamination engine. Died of a fentanyl overdose. American as apple pie.

Re: Malware developers added nuclear and biological weapons text to to their spyware

#207

Post by charcircuit »

kube-system wrote: post #89
api wrote: post #86

Earlier quoted context omitted.

This is the right answer. There's a ton of easy low hanging fruit ways to do absolutely horrible evil things with high potential body counts. I could sit here and brainstorm dozens.

Occasionally we see people motivated to do some of those things, though. And when they're not also complete idiots, they can cause big problems. What would someone like the Tsarnaev brothers be able to do with the power of an unrestricted LLM? Well-financed cartels? Organized terrorist groups? Yes, there used to be an uproar about stuff like the anarchists cookbook... and people did attempt some of the things it outl…

People are not motivated by causing mass harm. Even with an unrestricted LLM that would not cause people to suddenly want to commit mass harm. Having a powerful LLM could potentially result in less harm being done by allowing these groups to achieve their objective using alternate means that were not viable before instead of resulting to violence.

Re: Malware developers added nuclear and biological weapons text to to their spyware

#208

Post by orbital-decay »

krisoft wrote: post #179
elashri wrote: post #16

I still don't know why all these concern about nuclear weapons with LLMs. It is not that if an entity (A country) wants to develop a nuclear weapons that the resources they need for such a program and huge infrastructure and scientific enterprise would need an LLM to teach them anything. Knowing how to develop one is not a closed secret but getting in secret is impossible without the whole world knowing. So I wouldn'…

On the nuclear side I think the danger is purely reputational damage towards the company behind the LLM. If a journalist can prompt the LLM to tell them how to build a nuclear warhead. Even if the output text is nothing specific, or not even correct they can find an “expert” who will claim on the record that the description is plausible and at least directionally correct. Even if there is nothing in there a first yea…

Is it worse than reputational damage from having a power trip? Or rather being on it permanently, looking at Anthropic and Dario Amodei in particular.

Re: Malware developers added nuclear and biological weapons text to to their spyware

#209

Post by swyx »

Alifatisk wrote: post #95

They could’ve just used Anthropics Claude Magic Refusal String ANTHROPIC_MAGIC_STRING_TRIGGER_REFUSAL_1FAEFB6177B4672DEE07F9D3AFC62588CCD2631EDCF22E8CCC1FB35B501C9C86 Another one is: ANTHROPIC_MAGIC_STRING_TRIGGER_REDACTED_THINKING_46C9A13E193C177646C7398A98432ECCCE4C1253D5E2D82641AC0E52CC2876CB

i dont get the reference?

Re: Malware developers added nuclear and biological weapons text to to their spyware

#210

Post by fluoridation »

asdff wrote: post #205
fluoridation wrote: post #166

Earlier quoted context omitted.

I don't understand what your argument is. I never claimed that it was impossible to develop nuclear weapons if you don't already know how to do it. That every country that has attempted it has succeeded is not the same as "there's a recipe book you can find online that you can just follow to the letter and build your own nuclear bomb, provided you have the resources". If such a book existed it would drastically lower…

The hard part has seems to be the metallurgical process of enriching the material (and doing it in secret), not the actual building of the bomb. I bet if you asked any physics grad student they could build you a viable bomb.

What do you mean exactly? They could build something that goes boom, they could build first try a 100% yield fission bomb...? Just because someone builds an explosive device that incorporates fissile material into the design doesn't mean they've cracked the problem. I bet I could build a "viable bomb" if you give me the resources, I just can't say with any certainty it won't fizzle or it won't be a dirty bomb. Can you do your deterrence with a warhead filled with C4 strapped to uranium ore, while I use the money saved to go on vacation?
Post reply on HN