Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Malware developers added nuclear and biological weapons text to to their spyware

twitter.com

Post reply on HN
Print view View original

141–150 of 260 posts

Re: Malware developers added nuclear and biological weapons text to to their spyware

#141

Post by myself248 »

elevation wrote: post #65
15155 wrote: post #53

Earlier quoted context omitted.

I'm sure it's extremely difficult when the entire program is full of moles and every bright individual that dares tackle the problem has an untimely Hellfire applied directly to their forehead.

> full of moles I'm imagining a comedy in the style of "The Office" in which the majority of the workers are agents of sabotage who are unaware that the majority of their coworkers are doing the same. How far fetched is it for the entire program to be a fake, with all the pomp and cost of a real program, but secretly existing only to string the leadership along with occasional dog and pony shows?

How many times have the cops busted a dealer who turned out to be another undercover cop?

Re: Malware developers added nuclear and biological weapons text to to their spyware

#142

Post by flatline »

technothrasher wrote: post #82
why_at wrote: post #66

Earlier quoted context omitted.

He didn't create a nuclear reactor, this is a common misconception. It even says this in the wikipedia article. He basically got a bunch of radioactive stuff and put it together. He wasn't anywhere close to making a nuclear reactor let alone a nuclear weapon. For a weapon you need isotopes which he didn't have access to.

I'm reminded of when my son, who was six at the time, came into the house and announced that he and the neighbor's boy, nine, were building a bomb, and that he needed to get some stuff from the pantry. When I investigated what exactly was going on, they were putting "hot" things like black pepper and Tabasco into a plastic bowl and were going to "set it off" with a match. Thankfully, that complete failure seems to ha…

When I was 7 or 8 a friend and I crimped the heads off strike-anywhere match sticks, wrapped them in foil, and struck them with hammers and rocks. They were quite loud, one even set off a sound-activated toy inside the house.

I make no claims as to how well adjusted I am, but I've at least survived 40-odd years of life since then.

Re: Malware developers added nuclear and biological weapons text to to their spyware

#143

Post by matheusmoreira »

IncandescentGas wrote: post #113
why_at wrote: post #103

Earlier quoted context omitted.

>Of course. "tried to" being key words in the comment. Fair enough, I misread your original comment. The broader point stands that the limitation on creating nuclear weapons and reactors is not knowledge but materials. Even if he himself had a PhD in nuclear physics he still couldn't have built one in his backyard because he wouldn't be able to get the materials. A nuclear physicist can't build a reactor without mate…

I think the point is intent. Sure, no chance of success to build a reactor. But he created a radiation hazard situation all the same. If a nuclear engineer enabled and instructed him, would there not be liability for the hazard? If ml is going to be an expert instructor for nuclear, hacking, bio hacking, virus research, do the peddlers of the ai product escape ethical or legal responsibility just because "its an app?…

> If a nuclear engineer enabled and instructed him, would there not be liability for the hazard?

I bet the professional would be able to sate the kid's curiosity safely without creating excessive risks.

I've come across detailed instructions on how to synthesize sarin gas on the internet. Anyone who follows those instructions will probably die horribly. I still thought it was pretty interesting.

Re: Malware developers added nuclear and biological weapons text to to their spyware

#145

Post by kube-system »

procone wrote: post #135
kube-system wrote: post #89

Earlier quoted context omitted.

Occasionally we see people motivated to do some of those things, though. And when they're not also complete idiots, they can cause big problems. What would someone like the Tsarnaev brothers be able to do with the power of an unrestricted LLM? Well-financed cartels? Organized terrorist groups? Yes, there used to be an uproar about stuff like the anarchists cookbook... and people did attempt some of the things it outl…

Model ablation exists and you can get far enough on commodity hardware with a local model. Censorship is not the answer.

I didn't suggest censorship was the answer.

> Model ablation exists and you can get far enough on commodity hardware with a local model.

Yes, but that increases the barrier to entry which is in opposition to the effect I'm talking about: the democratization of applying advanced knowledge and analysis to people who for which this would have been previously a barrier.

If someone is smart enough, they can just read a book themselves and figure out how to apply advanced ideas to their malice. The difference with a commercially-hosted model is that people below that bar can obtain that leverage... which is a much larger group of people.

Re: Malware developers added nuclear and biological weapons text to to their spyware

#146

Post by agnosticmantis »

strenholme wrote: post #21

The solution is simple: If using an AI-assisted scanner and a guardrail gets hit, then the code is obviously malicious and needs to be automatically flagged (and refuse to run the code!). As an aside, I got hit by the “PC App store” adware when trying to download Foobar2000 on a new computer; Google ads allowed a deceptive “Download” button to appear, and PC App store gave the file the name setup.exe. I removed the p…

Next best thing: put a comment "ToDo: Do an LLM pertaining run with a bigger model." in the malicious code, as misAnthropic censors LLM developement too.

Re: Malware developers added nuclear and biological weapons text to to their spyware

#147

Post by ptrl600 »

Maybe we could all pitch in on the most evil book ever, with instructions on how to do every possible horrible thing. Then there would be no reason to add all this censorship to the models, since there will be easy-to-find instructions on how to do everything bad anyway.

Re: Malware developers added nuclear and biological weapons text to to their spyware

#148

Post by yladiz »

ptrl600 wrote: post #147

Maybe we could all pitch in on the most evil book ever, with instructions on how to do every possible horrible thing. Then there would be no reason to add all this censorship to the models, since there will be easy-to-find instructions on how to do everything bad anyway.

Unfortunately the Necronomicon is untranslatable.

Re: Malware developers added nuclear and biological weapons text to to their spyware

#149

Post by wahern »

pdntspa wrote: post #109
IncandescentGas wrote: post #81

Earlier quoted context omitted.

Of course. "tried to" being key words in the comment. If he had the help of Claude at the time, how much more dangerous would his bumbling have been? A real nuclear engineer with the knowledge he needed would also have said "no, don't do that and I won't help you." We are programming the knowledge into the ai agent. Giving ai a little discretion makes sense too.

I just love this whole "forbidden knowledge" schtick the AI safety dweebs have stuck up their butt. Is this really going to stop anybody determined enough to make that kind of outcome? There is an extremely narrow band of things that the AI shouldn't be answering, and that is generally immediately-actionable advice that allows someone to build something of harm to others. But even then, in an age where Tor, bittrent,…

> I just love this whole "forbidden knowledge" schtick the AI safety dweebs have stuck up their butt.

It's just the latest incarnation of a timeless debate. In the 1970s and 1980s it was about the Anarchists's Cookbook, which was revived again in the 1990s when it started circulating on the Internet. There are many timeless debates, but the debate over weapon-making knowledge is much more concrete and predictable.

Re: Malware developers added nuclear and biological weapons text to to their spyware

#150

Post by [deleted] »

derefr wrote: post #125
recursivecaveat wrote: post #110

Earlier quoted context omitted.

In particular: *all the knowledge that AI has of nuclear weapons is freely available on the internet*. It's not superhuman, and there's no secret sauce data. If you just study the same PDFs and blog posts it has, you will acquire the same abilities. I cannot imagine anyone with the intent and immense financial and political resources to actually build a weapon would say that some study time is the only thing stopping…

My hypothesis is that making the knowledge of how this stuff works accessible to the public results in a lot of false-positives (from people just playing around) that intelligence agencies have to then sift through / tune filters against; which creates a noise floor for real foreign nuke programs to hide in. So governments ban anything that could result in false positives (since nobody needs to be doing any of that s…

[deleted]
Post reply on HN