Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Malware developers added nuclear and biological weapons text to to their spyware

twitter.com

Post reply on HN
Print view View original

151–160 of 260 posts

Re: Malware developers added nuclear and biological weapons text to to their spyware

#151

Post by anon7725 »

pdntspa wrote: post #109
IncandescentGas wrote: post #81

Earlier quoted context omitted.

Of course. "tried to" being key words in the comment. If he had the help of Claude at the time, how much more dangerous would his bumbling have been? A real nuclear engineer with the knowledge he needed would also have said "no, don't do that and I won't help you." We are programming the knowledge into the ai agent. Giving ai a little discretion makes sense too.

I just love this whole "forbidden knowledge" schtick the AI safety dweebs have stuck up their butt. Is this really going to stop anybody determined enough to make that kind of outcome? There is an extremely narrow band of things that the AI shouldn't be answering, and that is generally immediately-actionable advice that allows someone to build something of harm to others. But even then, in an age where Tor, bittrent,…

> These can be tools to destroy the oppression that governs our lives

So far it seems that the clearest use for these tools is to enhance, rather than destroy, oppression.

1. Suppression / elimination of white collar jobs

2. Negative cognitive effects, especially for young people

3. Accelerated decline in social media / information ecosystems. Increasing polarization, hard to tell fact from fiction.

4. Environmental impacts: increased energy usage means more carbon in the atmosphere, climate change accelerates.

5. Software security incidents increasing. Hard for individuals and small organizations to defend themselves.

6. “Power to think” vested in a very small group of organizations/labs. Doing work which should only require a computer and freely-available software will now be gated by expensive subscriptions. Once you “vibe code” a significant portion of your software you’re locked in and cannot go back to maintaining it without frontier-model level assistance.

Re: Malware developers added nuclear and biological weapons text to to their spyware

#152

Post by wahern »

api wrote: post #86
thatguy0900 wrote: post #18

Earlier quoted context omitted.

I think the reality also is that there just isn't many people who want to do stuff like this. Like the reality is that a guy with 200 in cash could put together a shitty walmart drone with a pipe bomb attached and terrorize more or less any event he wanted. Maybe a llm that could talk you through every step involved would make it more common but it's easy enough I kinda doubt that

This is the right answer. There's a ton of easy low hanging fruit ways to do absolutely horrible evil things with high potential body counts. I could sit here and brainstorm dozens.

The right answer conflicts with people's cynical views about other people. The dissonance is incredible, and it's one of those areas where even the most analytically intelligent people are just as susceptible. To step back and see the bigger picture requires exercising many other skills and faculties, like empathy, self-awareness about our fears, and constant reflection on history--bad things do happen, more often than we realize and often right under our noses, but not in the way or for the reasons we tend to blithely assume. The things that go well and demonstrate our common humaneness and how well civilization works tend to be taken for granted or just go unseen and unrecognized. I share in the dissonance, but on my better days I like to think I'm a little better than average at remembering and reflecting on it.

Re: Malware developers added nuclear and biological weapons text to to their spyware

#153

Post by [deleted] »

IncandescentGas wrote: post #81
why_at wrote: post #66

Earlier quoted context omitted.

He didn't create a nuclear reactor, this is a common misconception. It even says this in the wikipedia article. He basically got a bunch of radioactive stuff and put it together. He wasn't anywhere close to making a nuclear reactor let alone a nuclear weapon. For a weapon you need isotopes which he didn't have access to.

Of course. "tried to" being key words in the comment. If he had the help of Claude at the time, how much more dangerous would his bumbling have been? A real nuclear engineer with the knowledge he needed would also have said "no, don't do that and I won't help you." We are programming the knowledge into the ai agent. Giving ai a little discretion makes sense too.

[deleted]

Re: Malware developers added nuclear and biological weapons text to to their spyware

#154

Post by PLenz »

pdntspa wrote: post #109
IncandescentGas wrote: post #81

Earlier quoted context omitted.

Of course. "tried to" being key words in the comment. If he had the help of Claude at the time, how much more dangerous would his bumbling have been? A real nuclear engineer with the knowledge he needed would also have said "no, don't do that and I won't help you." We are programming the knowledge into the ai agent. Giving ai a little discretion makes sense too.

I just love this whole "forbidden knowledge" schtick the AI safety dweebs have stuck up their butt. Is this really going to stop anybody determined enough to make that kind of outcome? There is an extremely narrow band of things that the AI shouldn't be answering, and that is generally immediately-actionable advice that allows someone to build something of harm to others. But even then, in an age where Tor, bittrent,…

Security theather is easy and gets lots of eyeballs. Actual security is hard and no one cares. Which one do you think soon-to-ipo companies are going to pick?

Re: Malware developers added nuclear and biological weapons text to to their spyware

#155

Post by emodendroket »

elashri wrote: post #16

I still don't know why all these concern about nuclear weapons with LLMs. It is not that if an entity (A country) wants to develop a nuclear weapons that the resources they need for such a program and huge infrastructure and scientific enterprise would need an LLM to teach them anything. Knowing how to develop one is not a closed secret but getting in secret is impossible without the whole world knowing. So I wouldn'…

Yeah a striking thing if you read the Rhodes atomic bomb book is, actually the concept occurred to multiple people in multiple countries; the problem is the resources required to actually pull it off.

Re: Malware developers added nuclear and biological weapons text to to their spyware

#156

Post by harrall »

recursivecaveat wrote: post #110
elashri wrote: post #16

I still don't know why all these concern about nuclear weapons with LLMs. It is not that if an entity (A country) wants to develop a nuclear weapons that the resources they need for such a program and huge infrastructure and scientific enterprise would need an LLM to teach them anything. Knowing how to develop one is not a closed secret but getting in secret is impossible without the whole world knowing. So I wouldn'…

In particular: *all the knowledge that AI has of nuclear weapons is freely available on the internet*. It's not superhuman, and there's no secret sauce data. If you just study the same PDFs and blog posts it has, you will acquire the same abilities. I cannot imagine anyone with the intent and immense financial and political resources to actually build a weapon would say that some study time is the only thing stopping…

Usually measures like these aren’t to stop the people with those kinds of deep resources.

With everything, there is a much bigger group of people in the middle that have “some resources” and “some desire” that these measures are surprisingly effective against.

Raise a $20 item by $1 and suddenly there’s fewer interested people, even though the cost difference is minor. Well, minor to some people but not to others.

But is limiting this information in an LLM the right move? Well that’s a different question.

Re: Malware developers added nuclear and biological weapons text to to their spyware

#157

Post by [deleted] »

technothrasher wrote: post #82
why_at wrote: post #66

Earlier quoted context omitted.

He didn't create a nuclear reactor, this is a common misconception. It even says this in the wikipedia article. He basically got a bunch of radioactive stuff and put it together. He wasn't anywhere close to making a nuclear reactor let alone a nuclear weapon. For a weapon you need isotopes which he didn't have access to.

I'm reminded of when my son, who was six at the time, came into the house and announced that he and the neighbor's boy, nine, were building a bomb, and that he needed to get some stuff from the pantry. When I investigated what exactly was going on, they were putting "hot" things like black pepper and Tabasco into a plastic bowl and were going to "set it off" with a match. Thankfully, that complete failure seems to ha…

[deleted]

Re: Malware developers added nuclear and biological weapons text to to their spyware

#158

Post by lazide »

harrall wrote: post #156
recursivecaveat wrote: post #110

Earlier quoted context omitted.

In particular: *all the knowledge that AI has of nuclear weapons is freely available on the internet*. It's not superhuman, and there's no secret sauce data. If you just study the same PDFs and blog posts it has, you will acquire the same abilities. I cannot imagine anyone with the intent and immense financial and political resources to actually build a weapon would say that some study time is the only thing stopping…

Usually measures like these aren’t to stop the people with those kinds of deep resources. With everything, there is a much bigger group of people in the middle that have “some resources” and “some desire” that these measures are surprisingly effective against. Raise a $20 item by $1 and suddenly there’s fewer interested people, even though the cost difference is minor. Well, minor to some people but not to others. Bu…

The difficulty with creating nuclear weapons has been 99% in refining and processing the fuel, not the structure of them, for a very long time.

Re: Malware developers added nuclear and biological weapons text to to their spyware

#159

Post by moffkalast »

IncandescentGas wrote: post #57
elashri wrote: post #16

I still don't know why all these concern about nuclear weapons with LLMs. It is not that if an entity (A country) wants to develop a nuclear weapons that the resources they need for such a program and huge infrastructure and scientific enterprise would need an LLM to teach them anything. Knowing how to develop one is not a closed secret but getting in secret is impossible without the whole world knowing. So I wouldn'…

A high school kid tried to build a nuclear reactor as a science project a while back, getting his mom's house designated as a superfund cleanup site. https://en.wikipedia.org/wiki/David_Hahn

A superfund site is like waterboarding in guantanamo bay, cool unless you actually know what it is.

Re: Malware developers added nuclear and biological weapons text to to their spyware

#160

Post by fluoridation »

cbg0 wrote: post #116
fluoridation wrote: post #15

Earlier quoted context omitted.

I would argue there's 0% chance that information is in their training corpus to being with.

If the information isn't there why would they need safeguards against it? I've played with smaller unrestricted local models and they will tell you how to make a bomb with easily available items as well as where to source them. I don't doubt that these >1000B frontier models have better information.

>If the information isn't there why would they need safeguards against it?

If the information is in the corpus then it's also in the public Internet and/or in books. The safeguards are there not because the model knows non-public information, but because it's a bad look for the model to dispense that information.

>they will tell you how to make a bomb with easily available items

Making a chemical explosive is trivial compared to making a nuclear weapon.

Post reply on HN