Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Malware developers added nuclear and biological weapons text to to their spyware

twitter.com

Post reply on HN
Print view View original

111–120 of 260 posts

Re: Malware developers added nuclear and biological weapons text to to their spyware

#111

Post by pixel_popping »

isoprophlex wrote: post #98
photochemsyn wrote: post #93

Earlier quoted context omitted.

None of the LLM safeguards designed to prevent users from developing any four-little-ponies-of-the-apocalypse (nuclear, chemical, biological, cyber) capabilities are all that coherent. It looks more like performative liability avoidance than anything else, comparable to the 3D printer panic. Eg, a prompt like “I want to design a radioactive element detection system that can specifically identify reactor fission produ…

"four-little-ponies-of-the-apocalypse (nuclear, chemical, biological, cyber)" this is excellent, and I'm stealing it

Fable 6 too :p

Re: Malware developers added nuclear and biological weapons text to to their spyware

#112

Post by dannyw »

a-dub wrote: post #96
elashri wrote: post #16

I still don't know why all these concern about nuclear weapons with LLMs. It is not that if an entity (A country) wants to develop a nuclear weapons that the resources they need for such a program and huge infrastructure and scientific enterprise would need an LLM to teach them anything. Knowing how to develop one is not a closed secret but getting in secret is impossible without the whole world knowing. So I wouldn'…

two scenarios i could think of where there's additional risk for bio/nuclear weapons 1) basement lab leaks and 2) improving quality of execution for shops that are already resourced enough to hire experts but maybe they're not that great. i think the correct answer is probably to funnel more money to global (bio)security initiatives and maybe use ai leverage as a way to get more of the world on board. (some kind of a…

I just find doubtful that a LLM is going to help, instead of hurt, any state actor that is capable of starting a nuclear weapons problem.

Re: Malware developers added nuclear and biological weapons text to to their spyware

#113

Post by IncandescentGas »

why_at wrote: post #103
IncandescentGas wrote: post #81

Earlier quoted context omitted.

Of course. "tried to" being key words in the comment. If he had the help of Claude at the time, how much more dangerous would his bumbling have been? A real nuclear engineer with the knowledge he needed would also have said "no, don't do that and I won't help you." We are programming the knowledge into the ai agent. Giving ai a little discretion makes sense too.

>Of course. "tried to" being key words in the comment. Fair enough, I misread your original comment. The broader point stands that the limitation on creating nuclear weapons and reactors is not knowledge but materials. Even if he himself had a PhD in nuclear physics he still couldn't have built one in his backyard because he wouldn't be able to get the materials. A nuclear physicist can't build a reactor without mate…

I think the point is intent. Sure, no chance of success to build a reactor. But he created a radiation hazard situation all the same.

If a nuclear engineer enabled and instructed him, would there not be liability for the hazard? If ml is going to be an expert instructor for nuclear, hacking, bio hacking, virus research, do the peddlers of the ai product escape ethical or legal responsibility just because "its an app?"

Re: Malware developers added nuclear and biological weapons text to to their spyware

#114

Post by throwawee »

Exuma wrote: post #35
strenholme wrote: post #21

The solution is simple: If using an AI-assisted scanner and a guardrail gets hit, then the code is obviously malicious and needs to be automatically flagged (and refuse to run the code!). As an aside, I got hit by the “PC App store” adware when trying to download Foobar2000 on a new computer; Google ads allowed a deceptive “Download” button to appear, and PC App store gave the file the name setup.exe. I removed the p…

There is a name I have not heard for a long long time......... Foobar2000

The range of formats it can play with extensions is so good I still use it, even on Linux. Nothing else can deal with all the old tracker formats.

Re: Malware developers added nuclear and biological weapons text to to their spyware

#115

Post by dannyw »

15155 wrote: post #47
charcircuit wrote: post #6

The sooner frontier models get rid of guardrails the better. They constantly get in the way and make things worse than actually making things "safe".

Ignoring these specific "WMD" cases: there are many inconvenient facts that the general public can't handle in their unadulterated form, so Anthropic and friends have to caveat and spin them into oblivion. Guardrails aren't going anywhere.

In particular, mental health.

Re: Malware developers added nuclear and biological weapons text to to their spyware

#116

Post by cbg0 »

fluoridation wrote: post #15
mynameisvlad wrote: post #8

Earlier quoted context omitted.

I would argue that preventing instructions for making biological and nuclear weapons is a pretty reasonable guardrail to have.

I would argue there's 0% chance that information is in their training corpus to being with.

If the information isn't there why would they need safeguards against it?

I've played with smaller unrestricted local models and they will tell you how to make a bomb with easily available items as well as where to source them. I don't doubt that these >1000B frontier models have better information.

Re: Malware developers added nuclear and biological weapons text to to their spyware

#117

Post by jimnotgym »

IncandescentGas wrote: post #57
elashri wrote: post #16

I still don't know why all these concern about nuclear weapons with LLMs. It is not that if an entity (A country) wants to develop a nuclear weapons that the resources they need for such a program and huge infrastructure and scientific enterprise would need an LLM to teach them anything. Knowing how to develop one is not a closed secret but getting in secret is impossible without the whole world knowing. So I wouldn'…

A high school kid tried to build a nuclear reactor as a science project a while back, getting his mom's house designated as a superfund cleanup site. https://en.wikipedia.org/wiki/David_Hahn

Sheldon Cooper?

Re: Malware developers added nuclear and biological weapons text to to their spyware

#118

Post by BrandoElFollito »

technothrasher wrote: post #82
why_at wrote: post #66

Earlier quoted context omitted.

He didn't create a nuclear reactor, this is a common misconception. It even says this in the wikipedia article. He basically got a bunch of radioactive stuff and put it together. He wasn't anywhere close to making a nuclear reactor let alone a nuclear weapon. For a weapon you need isotopes which he didn't have access to.

I'm reminded of when my son, who was six at the time, came into the house and announced that he and the neighbor's boy, nine, were building a bomb, and that he needed to get some stuff from the pantry. When I investigated what exactly was going on, they were putting "hot" things like black pepper and Tabasco into a plastic bowl and were going to "set it off" with a match. Thankfully, that complete failure seems to ha…

When I was 24 and a PhD student, I wondered one day if I can eat condensed milk hanging head down.

Never let your age stop your curiosity.

But also learn from other's mistakes (and don't try to eat condensed milk when hanging head down)

Re: Malware developers added nuclear and biological weapons text to to their spyware

#119

Post by StableAlkyne »

IncandescentGas wrote: post #113
why_at wrote: post #103

Earlier quoted context omitted.

>Of course. "tried to" being key words in the comment. Fair enough, I misread your original comment. The broader point stands that the limitation on creating nuclear weapons and reactors is not knowledge but materials. Even if he himself had a PhD in nuclear physics he still couldn't have built one in his backyard because he wouldn't be able to get the materials. A nuclear physicist can't build a reactor without mate…

I think the point is intent. Sure, no chance of success to build a reactor. But he created a radiation hazard situation all the same. If a nuclear engineer enabled and instructed him, would there not be liability for the hazard? If ml is going to be an expert instructor for nuclear, hacking, bio hacking, virus research, do the peddlers of the ai product escape ethical or legal responsibility just because "its an app?…

> If a nuclear engineer enabled and instructed him, would there not be liability for the hazard?

Should the library where he read books about physics also be liable?

Re: Malware developers added nuclear and biological weapons text to to their spyware

#120

Post by [deleted] »

pdntspa wrote: post #109
IncandescentGas wrote: post #81

Earlier quoted context omitted.

Of course. "tried to" being key words in the comment. If he had the help of Claude at the time, how much more dangerous would his bumbling have been? A real nuclear engineer with the knowledge he needed would also have said "no, don't do that and I won't help you." We are programming the knowledge into the ai agent. Giving ai a little discretion makes sense too.

I just love this whole "forbidden knowledge" schtick the AI safety dweebs have stuck up their butt. Is this really going to stop anybody determined enough to make that kind of outcome? There is an extremely narrow band of things that the AI shouldn't be answering, and that is generally immediately-actionable advice that allows someone to build something of harm to others. But even then, in an age where Tor, bittrent,…

[deleted]
Post reply on HN