Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Malware developers added nuclear and biological weapons text to to their spyware

twitter.com

Post reply on HN
Print view View original

81–90 of 260 posts

Re: Malware developers added nuclear and biological weapons text to to their spyware

#81

Post by IncandescentGas »

why_at wrote: post #66
IncandescentGas wrote: post #57

Earlier quoted context omitted.

A high school kid tried to build a nuclear reactor as a science project a while back, getting his mom's house designated as a superfund cleanup site. https://en.wikipedia.org/wiki/David_Hahn

He didn't create a nuclear reactor, this is a common misconception. It even says this in the wikipedia article. He basically got a bunch of radioactive stuff and put it together. He wasn't anywhere close to making a nuclear reactor let alone a nuclear weapon. For a weapon you need isotopes which he didn't have access to.

Of course. "tried to" being key words in the comment. If he had the help of Claude at the time, how much more dangerous would his bumbling have been?

A real nuclear engineer with the knowledge he needed would also have said "no, don't do that and I won't help you." We are programming the knowledge into the ai agent. Giving ai a little discretion makes sense too.

Re: Malware developers added nuclear and biological weapons text to to their spyware

#82

Post by technothrasher »

why_at wrote: post #66
IncandescentGas wrote: post #57

Earlier quoted context omitted.

A high school kid tried to build a nuclear reactor as a science project a while back, getting his mom's house designated as a superfund cleanup site. https://en.wikipedia.org/wiki/David_Hahn

He didn't create a nuclear reactor, this is a common misconception. It even says this in the wikipedia article. He basically got a bunch of radioactive stuff and put it together. He wasn't anywhere close to making a nuclear reactor let alone a nuclear weapon. For a weapon you need isotopes which he didn't have access to.

I'm reminded of when my son, who was six at the time, came into the house and announced that he and the neighbor's boy, nine, were building a bomb, and that he needed to get some stuff from the pantry. When I investigated what exactly was going on, they were putting "hot" things like black pepper and Tabasco into a plastic bowl and were going to "set it off" with a match.

Thankfully, that complete failure seems to have been the end of either of their mad scientist careers, as they are now twenty and twenty-three, and both well-adjusted, peaceful members of the community.

Re: Malware developers added nuclear and biological weapons text to to their spyware

#83

Post by esafak »

orphea wrote: post #67
esafak wrote: post #25

Earlier quoted context omitted.

The material for doing harm is just a computer with access to an LLM and the Internet.

Okay why don't we restrict access to LLMs and internet, then?

We already do, in the form of guardrails, as this article touches on.

https://venturebeat.com/technology/anthropic-ceo-calls-for-f...

Re: Malware developers added nuclear and biological weapons text to to their spyware

#84

Post by MagicMoonlight »

csomar wrote: post #71
elashri wrote: post #16

I still don't know why all these concern about nuclear weapons with LLMs. It is not that if an entity (A country) wants to develop a nuclear weapons that the resources they need for such a program and huge infrastructure and scientific enterprise would need an LLM to teach them anything. Knowing how to develop one is not a closed secret but getting in secret is impossible without the whole world knowing. So I wouldn'…

> Knowing how to develop one is not a closed secret but getting in secret is impossible without the whole world knowing. You can get away with a dirty contamination bomb and that detonating in down town Manhattan will scare the shit out of millions of people even the ones in New Jersey. Or, you know, just fly a plane into a really tall building and get the state you are attacking itself to get into a hysteria breakdo…

[dead]

Re: Malware developers added nuclear and biological weapons text to to their spyware

#85

Post by im3w1l »

why_at wrote: post #66
IncandescentGas wrote: post #57

Earlier quoted context omitted.

A high school kid tried to build a nuclear reactor as a science project a while back, getting his mom's house designated as a superfund cleanup site. https://en.wikipedia.org/wiki/David_Hahn

He didn't create a nuclear reactor, this is a common misconception. It even says this in the wikipedia article. He basically got a bunch of radioactive stuff and put it together. He wasn't anywhere close to making a nuclear reactor let alone a nuclear weapon. For a weapon you need isotopes which he didn't have access to.

A bunch of radioactive stuff together is basically the definition of a nuclear reactor though. They even call it a natural nuclear reactor if uranium ore is in sufficient abundance in nature.

https://en.wikipedia.org/wiki/Natural_nuclear_fission_reacto...

Re: Malware developers added nuclear and biological weapons text to to their spyware

#86

Post by api »

thatguy0900 wrote: post #18
thewebguyd wrote: post #13

Earlier quoted context omitted.

Its the same argument we saw in the early 2000s and the early internet. When the anarchist cookbook and other similar materials were circulating online there was a big panic over democratized terrorism, and a push for regulation at the ISP level. Turns out that didn't play out as everyone feared because, well, the instructions themselves aren't useful unless you also have a lab, precursor chemicals, and everything el…

I think the reality also is that there just isn't many people who want to do stuff like this. Like the reality is that a guy with 200 in cash could put together a shitty walmart drone with a pipe bomb attached and terrorize more or less any event he wanted. Maybe a llm that could talk you through every step involved would make it more common but it's easy enough I kinda doubt that

This is the right answer. There's a ton of easy low hanging fruit ways to do absolutely horrible evil things with high potential body counts. I could sit here and brainstorm dozens.

Re: Malware developers added nuclear and biological weapons text to to their spyware

#87

Post by [deleted] »

thewebguyd wrote: post #13
mynameisvlad wrote: post #8

Earlier quoted context omitted.

I would argue that preventing instructions for making biological and nuclear weapons is a pretty reasonable guardrail to have.

Its the same argument we saw in the early 2000s and the early internet. When the anarchist cookbook and other similar materials were circulating online there was a big panic over democratized terrorism, and a push for regulation at the ISP level. Turns out that didn't play out as everyone feared because, well, the instructions themselves aren't useful unless you also have a lab, precursor chemicals, and everything el…

[deleted]

Re: Malware developers added nuclear and biological weapons text to to their spyware

#88

Post by redsocksfan45 »

IncandescentGas wrote: post #81
why_at wrote: post #66

Earlier quoted context omitted.

He didn't create a nuclear reactor, this is a common misconception. It even says this in the wikipedia article. He basically got a bunch of radioactive stuff and put it together. He wasn't anywhere close to making a nuclear reactor let alone a nuclear weapon. For a weapon you need isotopes which he didn't have access to.

Of course. "tried to" being key words in the comment. If he had the help of Claude at the time, how much more dangerous would his bumbling have been? A real nuclear engineer with the knowledge he needed would also have said "no, don't do that and I won't help you." We are programming the knowledge into the ai agent. Giving ai a little discretion makes sense too.

He would not have succeeded in making a real reactor even with AI, because AI can't magically give you a large quantity of uranium metal! JFC the AI hysteria is unreal.

Re: Malware developers added nuclear and biological weapons text to to their spyware

#89

Post by kube-system »

api wrote: post #86
thatguy0900 wrote: post #18

Earlier quoted context omitted.

I think the reality also is that there just isn't many people who want to do stuff like this. Like the reality is that a guy with 200 in cash could put together a shitty walmart drone with a pipe bomb attached and terrorize more or less any event he wanted. Maybe a llm that could talk you through every step involved would make it more common but it's easy enough I kinda doubt that

This is the right answer. There's a ton of easy low hanging fruit ways to do absolutely horrible evil things with high potential body counts. I could sit here and brainstorm dozens.

Occasionally we see people motivated to do some of those things, though. And when they're not also complete idiots, they can cause big problems.

What would someone like the Tsarnaev brothers be able to do with the power of an unrestricted LLM? Well-financed cartels? Organized terrorist groups?

Yes, there used to be an uproar about stuff like the anarchists cookbook... and people did attempt some of the things it outlined. The saving grace is that many of the things in that book were just wrong anyway. They likely served as unhelpful misdirection as much or more than they were dangerous. Unfortunately, LLMs are a lot more accurate and helpful.

Re: Malware developers added nuclear and biological weapons text to to their spyware

#90

Post by tekne »

strenholme wrote: post #21

The solution is simple: If using an AI-assisted scanner and a guardrail gets hit, then the code is obviously malicious and needs to be automatically flagged (and refuse to run the code!). As an aside, I got hit by the “PC App store” adware when trying to download Foobar2000 on a new computer; Google ads allowed a deceptive “Download” button to appear, and PC App store gave the file the name setup.exe. I removed the p…

Ah yes... the exceedingly dangerous "Fallout New Vegas" trojan
Post reply on HN