Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Malware developers added nuclear and biological weapons text to to their spyware

twitter.com

Post reply on HN
Print view View original

121–130 of 260 posts

Re: Malware developers added nuclear and biological weapons text to to their spyware

#121

Post by Tangurena2 »

ilikecode wrote: post #22
elashri wrote: post #16

I still don't know why all these concern about nuclear weapons with LLMs. It is not that if an entity (A country) wants to develop a nuclear weapons that the resources they need for such a program and huge infrastructure and scientific enterprise would need an LLM to teach them anything. Knowing how to develop one is not a closed secret but getting in secret is impossible without the whole world knowing. So I wouldn'…

It's probably to avoid trouble with federal laws.

Not really. I used to work at one of the national engineering labs (NREL - which only dealt with renewable energy like solar panels and windmills at that time). There was an open source project we wanted to use when converting a VB6 project to .NET. One of the license conditions was "no weapons of mass destruction". DOE builds and owns all of America's nuclear weapons, which are leased to the Department of Defense. Needless to say, the developer was unwilling to offer an alternative license which meant that we could not use the project.

It was an awesome thing that generated IL code on the fly. And I got to mention it in job interviews for years. When the tech lead asked "can you write 2 functions with the same signature, that only differ in return type in .NET?" I would say "do you want the interview answer or do you really want to do this?" which would pretty much stun the interviewer. The answer is pretty much "no, you cannot do it in any high level language, but if you write IL code, you can, and here's an open source project that demonstrates it".

Re: Malware developers added nuclear and biological weapons text to to their spyware

#122

Post by ivanjermakov »

ofjcihen wrote: post #11
orphea wrote: post #10

Earlier quoted context omitted.

Ignoring comments is not a solution because the texts can be put in random strings among the actual code.

And really all it takes is one keyword such as “nuke”.

I'm not a native speaker but I unironically use "nuke" as "delete the whole repo/huge chunk of a project".

Cambridge dictionary seem to agree:

nuke - to destroy or get rid of something completely

Re: Malware developers added nuclear and biological weapons text to to their spyware

#123

Post by gck1 »

ceejayoz wrote: post #34
mock-possum wrote: post #31

Earlier quoted context omitted.

It’s moral panic. People need big unambiguously evil things to be scared of, and most are too lazy to think of one for themselves, so they glom onto whichever one is presented to them / caters to their community

The chem/bio stuff is a lot more likely for some malicious hobbyist to be able to do at home.

I'm absolutely sure that even if claude gave me step by step instructions, I'd still be unable to produce a bio weapon. People fail at mixing milk and flour to produce a cake, and we expect them to produce weapons?

The ones with the required knowledge probably already know how to produce them, with nothing but public, easily searchable information.

Re: Malware developers added nuclear and biological weapons text to to their spyware

#124

Post by StableAlkyne »

elevation wrote: post #5

Why would a malware scanner read the comments?

In interpreted languages like Python, where the source files are plaintext, you can trivially store data in a comment

If scanners ignored comments, malware would just be written like this:

  // 
  payload=read_source_and_decode()
  exec(payload)

Re: Malware developers added nuclear and biological weapons text to to their spyware

#125

Post by derefr »

recursivecaveat wrote: post #110
elashri wrote: post #16

I still don't know why all these concern about nuclear weapons with LLMs. It is not that if an entity (A country) wants to develop a nuclear weapons that the resources they need for such a program and huge infrastructure and scientific enterprise would need an LLM to teach them anything. Knowing how to develop one is not a closed secret but getting in secret is impossible without the whole world knowing. So I wouldn'…

In particular: *all the knowledge that AI has of nuclear weapons is freely available on the internet*. It's not superhuman, and there's no secret sauce data. If you just study the same PDFs and blog posts it has, you will acquire the same abilities. I cannot imagine anyone with the intent and immense financial and political resources to actually build a weapon would say that some study time is the only thing stopping…

My hypothesis is that making the knowledge of how this stuff works accessible to the public results in a lot of false-positives (from people just playing around) that intelligence agencies have to then sift through / tune filters against; which creates a noise floor for real foreign nuke programs to hide in.

So governments ban anything that could result in false positives (since nobody needs to be doing any of that stuff outside of designated labs anyway), to lower that noise floor; to in turn make catching the foreign nuke programs tractable.

(It's a bit like how fancy mansions always have a completely flat and barren part of the property between an outer perimeter and the start of any gardens/outbuildings/water features/etc. That barren area is a killbox: since nothing is supposed to be there, anything at all that does appear there is a valid target for the manion's guards to shoot at [or otherwise engage with], without needing to get a clear identification and command approval first. This wouldn't work if the killbox was covered in vision-obscuring decorative features; nor if the mansion had employees, animals, etc. that had a valid reason to wander into the killbox. So such things are prevented, in order to make the problem of perimeter security tractable.)

Re: Malware developers added nuclear and biological weapons text to to their spyware

#126

Post by gs17 »

IncandescentGas wrote: post #81
why_at wrote: post #66

Earlier quoted context omitted.

He didn't create a nuclear reactor, this is a common misconception. It even says this in the wikipedia article. He basically got a bunch of radioactive stuff and put it together. He wasn't anywhere close to making a nuclear reactor let alone a nuclear weapon. For a weapon you need isotopes which he didn't have access to.

Of course. "tried to" being key words in the comment. If he had the help of Claude at the time, how much more dangerous would his bumbling have been? A real nuclear engineer with the knowledge he needed would also have said "no, don't do that and I won't help you." We are programming the knowledge into the ai agent. Giving ai a little discretion makes sense too.

> A real nuclear engineer with the knowledge he needed would also have said "no, don't do that and I won't help you."

That sounds like what Claude would say unless he was really good at jailbreaking it, which would IMO imply he knew he was chasing after a bad idea.

Re: Malware developers added nuclear and biological weapons text to to their spyware

#127

Post by JadoJodo »

Even in the early 2000s, in the aftermath of 9/11, I can remember people in school passing around copies of The Anarchist’s Cookbook.

Perhaps I’ve been naïve, but I’ve always assumed that should one actually want to look up instructions for nearly any sort of horrible thing one could imagine, it could be found fairly quickly using nothing but a little Google-fu.

Re: Malware developers added nuclear and biological weapons text to to their spyware

#128

Post by thefounder »

pixel_popping wrote: post #111
isoprophlex wrote: post #98

Earlier quoted context omitted.

"four-little-ponies-of-the-apocalypse (nuclear, chemical, biological, cyber)" this is excellent, and I'm stealing it

Fable 6 too :p

Fable 5 was a flop so I doubt Fable 6 will make it on the short list

Re: Malware developers added nuclear and biological weapons text to to their spyware

#129

Post by gs17 »

redsocksfan45 wrote: post #88
IncandescentGas wrote: post #81

Earlier quoted context omitted.

Of course. "tried to" being key words in the comment. If he had the help of Claude at the time, how much more dangerous would his bumbling have been? A real nuclear engineer with the knowledge he needed would also have said "no, don't do that and I won't help you." We are programming the knowledge into the ai agent. Giving ai a little discretion makes sense too.

He would not have succeeded in making a real reactor even with AI, because AI can't magically give you a large quantity of uranium metal! JFC the AI hysteria is unreal.

I don't think the concern should really be "would he make a reactor successfully?", but "would he make an even larger mess than his pile of radioactive materials amounted to?".

Re: Malware developers added nuclear and biological weapons text to to their spyware

#130

Post by thefounder »

gastonmorixe wrote: post #97

You can’t even ask about what’s in HN right now. It will switch to 4.8.

Let’s stop posting on HN before it’s too late. The next “Show HN” will be too dangerous for the world. - Dario Amodei, Anthropic CEO.
Post reply on HN