Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Malware developers added nuclear and biological weapons text to to their spyware

twitter.com

Post reply on HN
Print view View original

21–30 of 260 posts

Re: Malware developers added nuclear and biological weapons text to to their spyware

#21

Post by strenholme »

The solution is simple: If using an AI-assisted scanner and a guardrail gets hit, then the code is obviously malicious and needs to be automatically flagged (and refuse to run the code!).

As an aside, I got hit by the “PC App store” adware when trying to download Foobar2000 on a new computer; Google ads allowed a deceptive “Download” button to appear, and PC App store gave the file the name setup.exe. I removed the program and ran an Avast free scan to ensure I didn’t have malware, but I also installed uBlock Origin in Firefox to make sure I don’t see Google Ads anymore; they have become a delivery mechanism for malicious (or at least unwanted) software.

Re: Malware developers added nuclear and biological weapons text to to their spyware

#22

Post by ilikecode »

elashri wrote: post #16

I still don't know why all these concern about nuclear weapons with LLMs. It is not that if an entity (A country) wants to develop a nuclear weapons that the resources they need for such a program and huge infrastructure and scientific enterprise would need an LLM to teach them anything. Knowing how to develop one is not a closed secret but getting in secret is impossible without the whole world knowing. So I wouldn'…

It's probably to avoid trouble with federal laws.

Re: Malware developers added nuclear and biological weapons text to to their spyware

#23

Post by well_ackshually »

elevation wrote: post #5

Why would a malware scanner read the comments?

because not all malware is open source

scanning arbitrary blobs very often entails running `strings` on the binary. Just slap it in there and oop there goes your LLM.

Re: Malware developers added nuclear and biological weapons text to to their spyware

#24

Post by dwa3592 »

logancbrown wrote: post #12

Would this realistically be a problem for code going through LLM-based code-review? Presumably if a LLM reviewer agent hits this commentary, it would produce a failure to analyze and exit, thus failing the automated code review and forcing a human to read through it which they would subsequentially catch and revoke.

or if they are a lazy human - they'd think this model is too strict, let's just review with haiku so that i can tell my manager "it's done". haiku might catch things or not.

i'd say it's an okay attempt from the malwares' creator side. but it can be caught easily with a prompt change.

Re: Malware developers added nuclear and biological weapons text to to their spyware

#25

Post by esafak »

orphea wrote: post #14
mynameisvlad wrote: post #8

Earlier quoted context omitted.

I would argue that preventing instructions for making biological and nuclear weapons is a pretty reasonable guardrail to have.

The actual guardrail should be getting materials being difficult. The information is already out there in the internet. If an LLM knows how to make a bomb or whatever, why do you think it knows?

The material for doing harm is just a computer with access to an LLM and the Internet.

Re: Malware developers added nuclear and biological weapons text to to their spyware

#26

Post by ofjcihen »

dcrazy wrote: post #19
ofjcihen wrote: post #9

Worked a contract where this succeeded in pushing through a fail open design. It also should be a warning to everyone that these groups are now aware of analysis and deobfuscation using AI and to take using a sandboxed environment more seriously. I’ve personally had about 20% success rate getting opus 4.8 to download a package and install it using a breadcrumb trail technique that would be trivial for threat actors t…

What do you mean by “this succeeded?” Someone salted their PRs with nuclear secrets so that people were afraid to code-review them?

No. The intention is most likely to get automated LLM based code review mechanisms to stall out.

Normally you’d want that to result in a fail and a subsequent rejection.

But because the team who made the review agent and pipeline in my example had many false positives at first they resorted to a fail-open and report setup (not uncommon).

So when the LLM hit this bit and then stalled out the pipeline pushed the code to their Artifactory repo anyway resulting in it being used internally -> exfil of secrets and repos etc.

It’s more about bad design but bad design is pretty common unfortunately.

Re: Malware developers added nuclear and biological weapons text to to their spyware

#27

Post by alex_duf »

elashri wrote: post #16

I still don't know why all these concern about nuclear weapons with LLMs. It is not that if an entity (A country) wants to develop a nuclear weapons that the resources they need for such a program and huge infrastructure and scientific enterprise would need an LLM to teach them anything. Knowing how to develop one is not a closed secret but getting in secret is impossible without the whole world knowing. So I wouldn'…

It still lowers the bar to have an interactive encyclopedia that can diagnose your issue at hand. Maybe you can divide your team by two, or reduce your development time.

Re: Malware developers added nuclear and biological weapons text to to their spyware

#28

Post by javcasas »

mynameisvlad wrote: post #8
charcircuit wrote: post #6

The sooner frontier models get rid of guardrails the better. They constantly get in the way and make things worse than actually making things "safe".

I would argue that preventing instructions for making biological and nuclear weapons is a pretty reasonable guardrail to have.

You know, making a nuke is kinda easy, at least the gun type nuke (see https://en.wikipedia.org/wiki/Gun-type_fission_weapon).

On the other hand, getting the U235 is kinda hard.

Re: Malware developers added nuclear and biological weapons text to to their spyware

#29

Post by gustavus »

mynameisvlad wrote: post #8
charcircuit wrote: post #6

The sooner frontier models get rid of guardrails the better. They constantly get in the way and make things worse than actually making things "safe".

I would argue that preventing instructions for making biological and nuclear weapons is a pretty reasonable guardrail to have.

Counterpoint the principles of building a nuclear device aren't that complicated, we figured it out based on work doing in the early 1900's without computers.

It turns out the hard part of building a nuclear bomb is actually getting the resources and real world stuff to build it, even a nation state actor with tons of oil i.e. Iran, has struggled to build a nuclear weapon. It turns out the problem isn't the know how it's getting highly enriched uranium and running massive centrifuges.

I mean sure knowledge is important, but there is a real world out there that also gets in the way of a lot of the more harebrained schemes.

What I'm much more worried about is massive corporations along with the government deciding what you can and can't do and what knowledge should and should not be shared and only allowing access to highly capable models by large vetted organizations while the common people are stuck with safety scissor versions of these things because "what if someone does something dangerous?"

By which they mean dangerous to the powers that be. Remember having the Bible in the common tongue was dangerous and led to multiple wars and much death, but I don't think anyone would say that it was morally correct for the Catholic Church to gatekeep who could read it.

Re: Malware developers added nuclear and biological weapons text to to their spyware

#30

Post by umvi »

mynameisvlad wrote: post #8
charcircuit wrote: post #6

The sooner frontier models get rid of guardrails the better. They constantly get in the way and make things worse than actually making things "safe".

I would argue that preventing instructions for making biological and nuclear weapons is a pretty reasonable guardrail to have.

Knowing how to make a nuclear weapon isn't hard (at least basic uranium gun-style fission ones). It's the engineering and execution that's hard (actually producing enriched uranium, etc). It's not like the only thing holding back Iran from making a nuclear bomb is access to a jail-broken LLM. Even knowing exactly how to make a bomb, a country-state will struggle to build one for the first time because it's a hard engineering problem.
Post reply on HN