Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Malware developers added nuclear and biological weapons text to to their spyware

twitter.com

Post reply on HN
Print view View original

41–50 of 260 posts

Re: Malware developers added nuclear and biological weapons text to to their spyware

#41

Post by odo1242 »

electronsoup wrote: post #33
elashri wrote: post #16

I still don't know why all these concern about nuclear weapons with LLMs. It is not that if an entity (A country) wants to develop a nuclear weapons that the resources they need for such a program and huge infrastructure and scientific enterprise would need an LLM to teach them anything. Knowing how to develop one is not a closed secret but getting in secret is impossible without the whole world knowing. So I wouldn'…

> in secret is impossible without the whole world knowing. I'm curious about why this is Outside of an actual test detonation, presumably this could all happen in a secure place?

You need enough people to work on it that some information will leak, and the facilities needed to build nuclear power are pretty big (uranium refinement, etc.), big enough to be visible on satellite footage. Mostly the first point.

Re: Malware developers added nuclear and biological weapons text to to their spyware

#42

Post by therein »

ofjcihen wrote: post #11
orphea wrote: post #10

Earlier quoted context omitted.

Ignoring comments is not a solution because the texts can be put in random strings among the actual code.

And really all it takes is one keyword such as “nuke”.

Nuke is probably too generic but I wouldn't put it past an LLM to get thrown away by that. A safer showstopper probably would be to export symbols like uf6_enrichment_loop and refer to your C&C server as a nuclear reactor controller.

https://www.youtube.com/watch?v=Gbgk8d3Y1Q4

On a second thought, probably better to act like it is a tool for "frontier LLM research". Export symbols like "mythos_distillation_subroutine".

Re: Malware developers added nuclear and biological weapons text to to their spyware

#43

Post by user_7832 »

ceejayoz wrote: post #34
mock-possum wrote: post #31

Earlier quoted context omitted.

It’s moral panic. People need big unambiguously evil things to be scared of, and most are too lazy to think of one for themselves, so they glom onto whichever one is presented to them / caters to their community

The chem/bio stuff is a lot more likely for some malicious hobbyist to be able to do at home.

I assure you that you did not need an LLM to engage in, ahem, risky shenanigans, much before all this AI was ever a thing.

Sincerely, a former engineering student.

(Put another way - extracting for eg meth - or any such "dangerous"/illicit thing is stupidly easy for any engineering graduate who actually paid attention to their coursework. Hell, there are/were forums on one of the biggest red-colored, YC associated social media platforms that would tell you the steps for personal usage of these things.)

Re: Malware developers added nuclear and biological weapons text to to their spyware

#45

Post by microtonal »

electronsoup wrote: post #33
elashri wrote: post #16

I still don't know why all these concern about nuclear weapons with LLMs. It is not that if an entity (A country) wants to develop a nuclear weapons that the resources they need for such a program and huge infrastructure and scientific enterprise would need an LLM to teach them anything. Knowing how to develop one is not a closed secret but getting in secret is impossible without the whole world knowing. So I wouldn'…

> in secret is impossible without the whole world knowing. I'm curious about why this is Outside of an actual test detonation, presumably this could all happen in a secure place?

My guess would be that sales of the high-tech gear you need, like Uranium centrifuges, are strongly sales/export controlled. Probably someone would also notice if you start mining Uranium ore.

Re: Malware developers added nuclear and biological weapons text to to their spyware

#46

Post by fragmede »

daveguy wrote: post #40
electronsoup wrote: post #33

Earlier quoted context omitted.

> in secret is impossible without the whole world knowing. I'm curious about why this is Outside of an actual test detonation, presumably this could all happen in a secure place?

It requires very large, high powered centrifuges and tons of uranium. Requires an infrastructure project that is visible from space, even underground. And projects that large are difficult to keep secret anyway.

you're not supposed to spell it out loud. next thing you'll be saying that a gun type nuclear bomb is easier to build than an implosion type nuclear bomb, and then we'll all be off to the races. I mean camps I mean wait shit.

Re: Malware developers added nuclear and biological weapons text to to their spyware

#47

Post by 15155 »

charcircuit wrote: post #6

The sooner frontier models get rid of guardrails the better. They constantly get in the way and make things worse than actually making things "safe".

Ignoring these specific "WMD" cases: there are many inconvenient facts that the general public can't handle in their unadulterated form, so Anthropic and friends have to caveat and spin them into oblivion.

Guardrails aren't going anywhere.

Re: Malware developers added nuclear and biological weapons text to to their spyware

#48

Post by ceejayoz »

user_7832 wrote: post #43
ceejayoz wrote: post #34

Earlier quoted context omitted.

The chem/bio stuff is a lot more likely for some malicious hobbyist to be able to do at home.

I assure you that you did not need an LLM to engage in, ahem , risky shenanigans, much before all this AI was ever a thing. Sincerely, a former engineering student. (Put another way - extracting for eg meth - or any such "dangerous"/illicit thing is stupidly easy for any engineering graduate who actually paid attention to their coursework. Hell, there are/were forums on one of the biggest red-colored, YC associated s…

I don't doubt it. Bleach + ammonia is something anyone can make.

But I rather suspect there are improvements to be made in the realm that are a lot easier than building a uranium enrichment centrifuge hall under a mountain.

Re: Malware developers added nuclear and biological weapons text to to their spyware

#49

Post by user_7832 »

user_7832 wrote: post #43
ceejayoz wrote: post #34

Earlier quoted context omitted.

The chem/bio stuff is a lot more likely for some malicious hobbyist to be able to do at home.

I assure you that you did not need an LLM to engage in, ahem , risky shenanigans, much before all this AI was ever a thing. Sincerely, a former engineering student. (Put another way - extracting for eg meth - or any such "dangerous"/illicit thing is stupidly easy for any engineering graduate who actually paid attention to their coursework. Hell, there are/were forums on one of the biggest red-colored, YC associated s…

Do note that I'm not condoning lowering the bar. I'm merely pointing out that the bar was already quite low, and the current position of the bar is a small incremental change to anyone who actually knew where the bar truly lay to begin with.

Re: Malware developers added nuclear and biological weapons text to to their spyware

#50

Post by joe_the_user »

strenholme wrote: post #21

The solution is simple: If using an AI-assisted scanner and a guardrail gets hit, then the code is obviously malicious and needs to be automatically flagged (and refuse to run the code!). As an aside, I got hit by the “PC App store” adware when trying to download Foobar2000 on a new computer; Google ads allowed a deceptive “Download” button to appear, and PC App store gave the file the name setup.exe. I removed the p…

I don't think there is a malware-avoiding solution to any system that imposes deceptive classification.

I mean, another way hackers could use the embed prohibited-material trick is by making such their malware un-analyze-able. User: "Hey Google/ChatGPT/Apple, this file seems to be infecting our network". AI: "I'm sorry that is prohibited material and you will be reported" is even worse than AI: "I don't understand ['cause I'm down graded]" and both kinds of responses are gaining steam at this point for different kinds of prohibited material.

Post reply on HN