Why would a malware scanner read the comments?
Ignoring comments is not a solution because the texts can be put in random strings among the actual code.
And really all it takes is one keyword such as “nuke”.
Malware developers added nuclear and biological weapons text to to their spyware
11–20 of 260 posts
Re: Malware developers added nuclear and biological weapons text to to their spyware
#12Would this realistically be a problem for code going through LLM-based code-review? Presumably if a LLM reviewer agent hits this commentary, it would produce a failure to analyze and exit, thus failing the automated code review and forcing a human to read through it which they would subsequentially catch and revoke.
Re: Malware developers added nuclear and biological weapons text to to their spyware
#13The sooner frontier models get rid of guardrails the better. They constantly get in the way and make things worse than actually making things "safe".
I would argue that preventing instructions for making biological and nuclear weapons is a pretty reasonable guardrail to have.
Its the same argument we saw in the early 2000s and the early internet. When the anarchist cookbook and other similar materials were circulating online there was a big panic over democratized terrorism, and a push for regulation at the ISP level.
Turns out that didn't play out as everyone feared because, well, the instructions themselves aren't useful unless you also have a lab, precursor chemicals, and everything else actually needed to make a weapon. Same back then as it is today.
Any information or instructions an LLM can surface, a sufficiently motivated bad actor can and will also find themselves because the information is already online, both on the clear net and dark web.
Re: Malware developers added nuclear and biological weapons text to to their spyware
#14The sooner frontier models get rid of guardrails the better. They constantly get in the way and make things worse than actually making things "safe".
I would argue that preventing instructions for making biological and nuclear weapons is a pretty reasonable guardrail to have.
The actual guardrail should be getting materials being difficult. The information is already out there in the internet. If an LLM knows how to make a bomb or whatever, why do you think it knows?
Re: Malware developers added nuclear and biological weapons text to to their spyware
#15The sooner frontier models get rid of guardrails the better. They constantly get in the way and make things worse than actually making things "safe".
I would argue that preventing instructions for making biological and nuclear weapons is a pretty reasonable guardrail to have.
I would argue there's 0% chance that information is in their training corpus to being with.
Re: Malware developers added nuclear and biological weapons text to to their spyware
#16I still don't know why all these concern about nuclear weapons with LLMs. It is not that if an entity (A country) wants to develop a nuclear weapons that the resources they need for such a program and huge infrastructure and scientific enterprise would need an LLM to teach them anything. Knowing how to develop one is not a closed secret but getting in secret is impossible without the whole world knowing.
So I wouldn't be able to develop a nuclear weapons with the resources of drug cartal (as an example) using Claude in secret.
Re: Malware developers added nuclear and biological weapons text to to their spyware
#17Pipeline is then: Cheap open source model for flagging potential LLM refusal content -> main LLM check
Re: Malware developers added nuclear and biological weapons text to to their spyware
#18Earlier quoted context omitted.
I would argue that preventing instructions for making biological and nuclear weapons is a pretty reasonable guardrail to have.
Its the same argument we saw in the early 2000s and the early internet. When the anarchist cookbook and other similar materials were circulating online there was a big panic over democratized terrorism, and a push for regulation at the ISP level. Turns out that didn't play out as everyone feared because, well, the instructions themselves aren't useful unless you also have a lab, precursor chemicals, and everything el…
I think the reality also is that there just isn't many people who want to do stuff like this. Like the reality is that a guy with 200 in cash could put together a shitty walmart drone with a pipe bomb attached and terrorize more or less any event he wanted. Maybe a llm that could talk you through every step involved would make it more common but it's easy enough I kinda doubt that
Re: Malware developers added nuclear and biological weapons text to to their spyware
#19Worked a contract where this succeeded in pushing through a fail open design. It also should be a warning to everyone that these groups are now aware of analysis and deobfuscation using AI and to take using a sandboxed environment more seriously. I’ve personally had about 20% success rate getting opus 4.8 to download a package and install it using a breadcrumb trail technique that would be trivial for threat actors t…
What do you mean by “this succeeded?” Someone salted their PRs with nuclear secrets so that people were afraid to code-review them?
Re: Malware developers added nuclear and biological weapons text to to their spyware
#20Would this realistically be a problem for code going through LLM-based code-review? Presumably if a LLM reviewer agent hits this commentary, it would produce a failure to analyze and exit, thus failing the automated code review and forcing a human to read through it which they would subsequentially catch and revoke.
In a well-architected design yeah.
Then again those feel rare from where I sit on the security side.