Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Malware developers added nuclear and biological weapons text to to their spyware

twitter.com

Post reply on HN
Print view View original

101–110 of 260 posts

Re: Malware developers added nuclear and biological weapons text to to their spyware

#101

Post by frereubu »

IncandescentGas wrote: post #81
why_at wrote: post #66

Earlier quoted context omitted.

He didn't create a nuclear reactor, this is a common misconception. It even says this in the wikipedia article. He basically got a bunch of radioactive stuff and put it together. He wasn't anywhere close to making a nuclear reactor let alone a nuclear weapon. For a weapon you need isotopes which he didn't have access to.

Of course. "tried to" being key words in the comment. If he had the help of Claude at the time, how much more dangerous would his bumbling have been? A real nuclear engineer with the knowledge he needed would also have said "no, don't do that and I won't help you." We are programming the knowledge into the ai agent. Giving ai a little discretion makes sense too.

I think you're picking the wrong example. If I had some sticks, a bit of mud and a few leaves, whether or not I had Claude wouldn't make a difference to my ability to make a nuclear weapon. There are probably better examples of ways where unmediated AI might facilitate something horrible, although probably on a smaller scale.

Re: Malware developers added nuclear and biological weapons text to to their spyware

#102

Post by garyfirestorm »

redsocksfan45 wrote: post #88
IncandescentGas wrote: post #81

Earlier quoted context omitted.

Of course. "tried to" being key words in the comment. If he had the help of Claude at the time, how much more dangerous would his bumbling have been? A real nuclear engineer with the knowledge he needed would also have said "no, don't do that and I won't help you." We are programming the knowledge into the ai agent. Giving ai a little discretion makes sense too.

He would not have succeeded in making a real reactor even with AI, because AI can't magically give you a large quantity of uranium metal! JFC the AI hysteria is unreal.

prompt -> LLM -> flying car should be just around the corner guys!

Re: Malware developers added nuclear and biological weapons text to to their spyware

#103

Post by why_at »

IncandescentGas wrote: post #81
why_at wrote: post #66

Earlier quoted context omitted.

He didn't create a nuclear reactor, this is a common misconception. It even says this in the wikipedia article. He basically got a bunch of radioactive stuff and put it together. He wasn't anywhere close to making a nuclear reactor let alone a nuclear weapon. For a weapon you need isotopes which he didn't have access to.

Of course. "tried to" being key words in the comment. If he had the help of Claude at the time, how much more dangerous would his bumbling have been? A real nuclear engineer with the knowledge he needed would also have said "no, don't do that and I won't help you." We are programming the knowledge into the ai agent. Giving ai a little discretion makes sense too.

>Of course. "tried to" being key words in the comment.

Fair enough, I misread your original comment.

The broader point stands that the limitation on creating nuclear weapons and reactors is not knowledge but materials. Even if he himself had a PhD in nuclear physics he still couldn't have built one in his backyard because he wouldn't be able to get the materials. A nuclear physicist can't build a reactor without materials anymore than a pilot can fly without an airplane.

Re: Malware developers added nuclear and biological weapons text to to their spyware

#105

Post by cheraderama »

kirubakaran wrote: post #91
technothrasher wrote: post #82

Earlier quoted context omitted.

I'm reminded of when my son, who was six at the time, came into the house and announced that he and the neighbor's boy, nine, were building a bomb, and that he needed to get some stuff from the pantry. When I investigated what exactly was going on, they were putting "hot" things like black pepper and Tabasco into a plastic bowl and were going to "set it off" with a match. Thankfully, that complete failure seems to ha…

When I was 5 or so, I was convinced that if I dropped a bowl of hot water into a bucket of cold water, I'd get big explosion. That experiment yielding lukewarm water ended my mad scientist career.

You should have collided water with antiwater.

Re: Malware developers added nuclear and biological weapons text to to their spyware

#106

Post by IncandescentGas »

redsocksfan45 wrote: post #88
IncandescentGas wrote: post #81

Earlier quoted context omitted.

Of course. "tried to" being key words in the comment. If he had the help of Claude at the time, how much more dangerous would his bumbling have been? A real nuclear engineer with the knowledge he needed would also have said "no, don't do that and I won't help you." We are programming the knowledge into the ai agent. Giving ai a little discretion makes sense too.

He would not have succeeded in making a real reactor even with AI, because AI can't magically give you a large quantity of uranium metal! JFC the AI hysteria is unreal.

> succeeded in making a real reactor

The concern here is not if an amateur attempt to make a reactor, hack a bank, bioengineer a medicine/poison is successful or not. Interactive and instructive access to some forms of knowledge used to come with discretion along side instruction.

Yes, perhaps your swearing at me in this context is a little hysterical

Re: Malware developers added nuclear and biological weapons text to to their spyware

#107

Post by why_at »

im3w1l wrote: post #85
why_at wrote: post #66

Earlier quoted context omitted.

He didn't create a nuclear reactor, this is a common misconception. It even says this in the wikipedia article. He basically got a bunch of radioactive stuff and put it together. He wasn't anywhere close to making a nuclear reactor let alone a nuclear weapon. For a weapon you need isotopes which he didn't have access to.

A bunch of radioactive stuff together is basically the definition of a nuclear reactor though. They even call it a natural nuclear reactor if uranium ore is in sufficient abundance in nature. https://en.wikipedia.org/wiki/Natural_nuclear_fission_reacto...

>A bunch of radioactive stuff together is basically the definition of a nuclear reactor though.

It really isn't.

A pile of radioactive waste isn't a reactor. Marie Curie's notes are famously contaminated with radioactive materials but they aren't a reactor. This is about as close as the boy scout got.

The Oklo fossil reactor is unique because it happened to form in the right circumstances to produce a fission chain reaction, which does make it a reactor. Not every uranium mine is a reactor, in fact this is the only one known.

Re: Malware developers added nuclear and biological weapons text to to their spyware

#108

Post by RIMR »

elashri wrote: post #16

I still don't know why all these concern about nuclear weapons with LLMs. It is not that if an entity (A country) wants to develop a nuclear weapons that the resources they need for such a program and huge infrastructure and scientific enterprise would need an LLM to teach them anything. Knowing how to develop one is not a closed secret but getting in secret is impossible without the whole world knowing. So I wouldn'…

I mean, the information is out there. The people who really want it already have it. It's not some massive secret. It really doesn't matter if Claude can or can't tell you how to build a nuclear bomb, because people already know how to do it.

The problem is that you need the power of a state or a massive corporation to come anywhere close to getting the materials to make a nuclear bomb. Knowledge of how to make a nuke isn't the threat.

If AI is a threat at all here, it would be in figuring out a simpler way to make a nuclear bomb, but that is highly theoretical, so what exactly are we putting up guardrails to protect against?

Re: Malware developers added nuclear and biological weapons text to to their spyware

#109

Post by pdntspa »

IncandescentGas wrote: post #81
why_at wrote: post #66

Earlier quoted context omitted.

He didn't create a nuclear reactor, this is a common misconception. It even says this in the wikipedia article. He basically got a bunch of radioactive stuff and put it together. He wasn't anywhere close to making a nuclear reactor let alone a nuclear weapon. For a weapon you need isotopes which he didn't have access to.

Of course. "tried to" being key words in the comment. If he had the help of Claude at the time, how much more dangerous would his bumbling have been? A real nuclear engineer with the knowledge he needed would also have said "no, don't do that and I won't help you." We are programming the knowledge into the ai agent. Giving ai a little discretion makes sense too.

I just love this whole "forbidden knowledge" schtick the AI safety dweebs have stuck up their butt. Is this really going to stop anybody determined enough to make that kind of outcome?

There is an extremely narrow band of things that the AI shouldn't be answering, and that is generally immediately-actionable advice that allows someone to build something of harm to others. But even then, in an age where Tor, bittrent, i2p, abliterated local models, etc are freely available, let alone numerous books and online resources, is there even a point? Is it worth fully compromising the principles of free agency to an increasingly oppressed populace?

But instead of that we are handing the keys to regressive and repressive governments to order the suppression of any knowledge they deem inconvenient. I really doubt anyone is going to take a principled stance when the company's party minders threaten local staff with a rubber hose or incarceration.

I'm sure China et al are already doing this.

For the past 30-40 years humanity has received an incredible gift in these sand-powered thinking brainboxes. A gift that allows the common man to empower himself with a force multiplier towards his own success, and now access to superintelligence the likes of which few have ever seen. These can be tools to destroy the oppression that governs our lives from foolhardy, greedy, bootlicking control freaks. And here we are squandering it.

Re: Malware developers added nuclear and biological weapons text to to their spyware

#110

Post by recursivecaveat »

elashri wrote: post #16

I still don't know why all these concern about nuclear weapons with LLMs. It is not that if an entity (A country) wants to develop a nuclear weapons that the resources they need for such a program and huge infrastructure and scientific enterprise would need an LLM to teach them anything. Knowing how to develop one is not a closed secret but getting in secret is impossible without the whole world knowing. So I wouldn'…

In particular: *all the knowledge that AI has of nuclear weapons is freely available on the internet*. It's not superhuman, and there's no secret sauce data. If you just study the same PDFs and blog posts it has, you will acquire the same abilities. I cannot imagine anyone with the intent and immense financial and political resources to actually build a weapon would say that some study time is the only thing stopping them from detonating a nuke.

It is pretty convenient for the labs to frame the conversation around this though, since it is easy to address, very few paying customers are rejected, and sounds scary (so surely the less scary sounding stuff must be solved right?)

Post reply on HN