Computer, make nuclear reactor. No mistakes.
Malware developers added nuclear and biological weapons text to to their spyware
161–170 of 260 posts
Re: Malware developers added nuclear and biological weapons text to to their spyware
#162Earlier quoted context omitted.
> A real nuclear engineer with the knowledge he needed would also have said "no, don't do that and I won't help you." That sounds like what Claude would say unless he was really good at jailbreaking it, which would IMO imply he knew he was chasing after a bad idea.
Right, which is exactly what elashri is objecting to. elashri said "Why do LLMs have restrictions on nuclear science", and IncandescentGas was explaining why they think those guardrails are a good idea. You're just agreeing with them.
Oh, I missed the word "also". Thanks for pointing it out!
Re: Malware developers added nuclear and biological weapons text to to their spyware
#163Earlier quoted context omitted.
He would not have succeeded in making a real reactor even with AI, because AI can't magically give you a large quantity of uranium metal! JFC the AI hysteria is unreal.
I don't think the concern should really be "would he make a reactor successfully?", but "would he make an even larger mess than his pile of radioactive materials amounted to?".
This just seems like a not great example to make that point though. Since whatever Claude tells the kid looking to build a reactor or even bomb is almost certainly going to be more grounded and professional than:
Step 1. Obtain pliers
Step 2. Obtain 300 discarded smoke detectors
Step 3. Start yanking!
You'd maybe get a sketchy but near-viable plan that could be dangerous if asked for a dirty bomb, but there the danger would more be the conventional explosives and not where to source radioisotopes, as it was already common knowledge that most residential smoke detectors contained americium until recently.
Re: Malware developers added nuclear and biological weapons text to to their spyware
#164I still don't know why all these concern about nuclear weapons with LLMs. It is not that if an entity (A country) wants to develop a nuclear weapons that the resources they need for such a program and huge infrastructure and scientific enterprise would need an LLM to teach them anything. Knowing how to develop one is not a closed secret but getting in secret is impossible without the whole world knowing. So I wouldn'…
The only hard thing about nuclear weapons is getting the radioactive material. By the time you get your bachelors degree, every nuclear engineering or physics student knows enough of how and why nukes work. Every nation that built a gun-type device successfully made theirs on their first attempt. Implosion takes some engineering, trial & error.
If I understand right, the hard part is purifying the radioactive material. Even if you have access to a uranium mine, there's a lot of work to filter the U-235 from the U-238 or to breed it into plutonium.
It's even harder if you start with other sources. But if you could figure out filtering it, a cubic kilometer of sea water should be enough for a bomb.
Re: Malware developers added nuclear and biological weapons text to to their spyware
#165Earlier quoted context omitted.
Usually measures like these aren’t to stop the people with those kinds of deep resources. With everything, there is a much bigger group of people in the middle that have “some resources” and “some desire” that these measures are surprisingly effective against. Raise a $20 item by $1 and suddenly there’s fewer interested people, even though the cost difference is minor. Well, minor to some people but not to others. Bu…
The difficulty with creating nuclear weapons has been 99% in refining and processing the fuel, not the structure of them, for a very long time.
True for fission bombs. Less true for fusion bombs. The principal makeup and manufacturing of fusion device parts like tampers are still unknown to the public. Having a supply of HEU does not tell you how to assemble a functional triple stage device or how to utilize tritium, an isotope that measurably decreases in purity by the day.
Re: Malware developers added nuclear and biological weapons text to to their spyware
#166Earlier quoted context omitted.
Wikipedia contains the high-level notions of how to make these things, not the details of how to solve the engineering challenges such as achieving supercriticality. You won't find that on any publicly disseminated document, you'll just have to figure it out by running your own nuclear development program.
It seems like every country that has been "allowed" to use nuclear weapons has figured it out though. It isn't like there are any that set off on this course and failed. AFAIK they all pretty much succeeded except Iran, probably because of all the blowing up of enrichment facilities. South Africa pulled it off. Israel pulled it off. North Korea pulled it off. India and Pakistan both pulled it off. Seems like anyone c…
I don't understand what your argument is. I never claimed that it was impossible to develop nuclear weapons if you don't already know how to do it. That every country that has attempted it has succeeded is not the same as "there's a recipe book you can find online that you can just follow to the letter and build your own nuclear bomb, provided you have the resources". If such a book existed it would drastically lower the barrier to build a nuclear bomb, because you could skip the science part and just follow the recipe, certain that it would work. To be clear, such books exist for drug manufacture; they exist neither for semiconductor manufacture nor for WMD manufacture.
Re: Malware developers added nuclear and biological weapons text to to their spyware
#167Earlier quoted context omitted.
>Of course. "tried to" being key words in the comment. Fair enough, I misread your original comment. The broader point stands that the limitation on creating nuclear weapons and reactors is not knowledge but materials. Even if he himself had a PhD in nuclear physics he still couldn't have built one in his backyard because he wouldn't be able to get the materials. A nuclear physicist can't build a reactor without mate…
I think the point is intent. Sure, no chance of success to build a reactor. But he created a radiation hazard situation all the same. If a nuclear engineer enabled and instructed him, would there not be liability for the hazard? If ml is going to be an expert instructor for nuclear, hacking, bio hacking, virus research, do the peddlers of the ai product escape ethical or legal responsibility just because "its an app?…
I agree LLMs can be harmful and that the companies behind them should be held liable to some extent, for example the recent news with Google being held responsible for their AI's defamation.[1]
This is a pretty different argument though. The comment that started this thread was talking about LLMs making potentially dangerous knowledge more available to bad actors, now we're talking about LLMs giving personally harmful advice.
You asked:
>If he had the help of Claude at the time, how much more dangerous would his bumbling have been?
Probably less? Even if you removed all the guardrails from Claude it would've likely told him his reactor plan wouldn't work and that he would have a high chance of poisoning himself and the environment.
Re: Malware developers added nuclear and biological weapons text to to their spyware
#168Earlier quoted context omitted.
The difficulty with creating nuclear weapons has been 99% in refining and processing the fuel, not the structure of them, for a very long time.
True for fission bombs. Less true for fusion bombs. The principal makeup and manufacturing of fusion device parts like tampers are still unknown to the public. Having a supply of HEU does not tell you how to assemble a functional triple stage device or how to utilize tritium, an isotope that measurably decreases in purity by the day.
[flagged]
Re: Malware developers added nuclear and biological weapons text to to their spyware
#169Earlier quoted context omitted.
The only hard thing about nuclear weapons is getting the radioactive material. By the time you get your bachelors degree, every nuclear engineering or physics student knows enough of how and why nukes work. Every nation that built a gun-type device successfully made theirs on their first attempt. Implosion takes some engineering, trial & error.
If I understand right, the hard part is purifying the radioactive material. Even if you have access to a uranium mine, there's a lot of work to filter the U-235 from the U-238 or to breed it into plutonium. It's even harder if you start with other sources. But if you could figure out filtering it, a cubic kilometer of sea water should be enough for a bomb.
Uranium is not even that rare, it's just that when chemistry fails at separating atoms, you have to use physics, and 3 ~proton~ (EDIT: neutron) masses is very little to work with
Re: Malware developers added nuclear and biological weapons text to to their spyware
#170Earlier quoted context omitted.
A bunch of radioactive stuff together is basically the definition of a nuclear reactor though. They even call it a natural nuclear reactor if uranium ore is in sufficient abundance in nature. https://en.wikipedia.org/wiki/Natural_nuclear_fission_reacto...
>A bunch of radioactive stuff together is basically the definition of a nuclear reactor though. It really isn't. A pile of radioactive waste isn't a reactor. Marie Curie's notes are famously contaminated with radioactive materials but they aren't a reactor. This is about as close as the boy scout got. The Oklo fossil reactor is unique because it happened to form in the right circumstances to produce a fission chain r…
Indeed. I said a bunch and I meant a bunch. Trace amounts is not a bunch.