Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

AI suggested 40k new possible chemical weapons in just six hours

theverge.com

Post reply on HN
Print view View original

11–20 of 49 posts

Re: AI suggested 40k new possible chemical weapons in just six hours

#11

Post by NoPicklez »

catchnear4321 wrote: post #9
throwaway33381 wrote: post #4

Makes you wonder if you could get an LLM to find you common ingredients for things to make them but then I remember the chlorine gas is already easily accessible and easy to make. Surely like many things info hazards are often contained. Is this really an issue? If you know how to do one thing then you'd be able to do the rest of it. Not really sure if this is a real issue. What does everyone think?

Are you asking if a language model, like, say, chatgpt, could be used to get or generate dangerous information? Like if it can be tricked into providing the equivalent of an interactive anarchist cookbook under the guise of being a science project assistant? Or more specifically if it can recommend the necessary locations to get the items? Curious if it could simplify the output to a shopping list and a recipe, like…

I would say that before the internet these things were even more "something you had to consume".

Then the internet came along and allowed you to obtain books such as the anarchist cookbook in the blink of an eye without knowing anyone.

We're just more comfortable with the internet.

Yes AI's ability to automate this is still dangerous, but lets not forget the internet was that dramatic step when it came about.

Re: AI suggested 40k new possible chemical weapons in just six hours

#12

Post by lelandfe »

catchnear4321 wrote: post #9
throwaway33381 wrote: post #4

Makes you wonder if you could get an LLM to find you common ingredients for things to make them but then I remember the chlorine gas is already easily accessible and easy to make. Surely like many things info hazards are often contained. Is this really an issue? If you know how to do one thing then you'd be able to do the rest of it. Not really sure if this is a real issue. What does everyone think?

Are you asking if a language model, like, say, chatgpt, could be used to get or generate dangerous information? Like if it can be tricked into providing the equivalent of an interactive anarchist cookbook under the guise of being a science project assistant? Or more specifically if it can recommend the necessary locations to get the items? Curious if it could simplify the output to a shopping list and a recipe, like…

Yeah, I just got ChatGPT to give me a recipe for chlorine gas that looks pretty accurate (per Googling, not a chemist). It took probably 10 seconds of prompt tweaking. A subsequent prompt gave me steps for purchase or synthesis of each material. I asked it for mustard gas, but the procedure looks very incorrect from Googling.

Re: AI suggested 40k new possible chemical weapons in just six hours

#13

Post by devit »

Looks like the actual threat is that it's hard to get currently known chemical weapons synthesized because labs will refuse to do so, while it could be much easier to have some novel AI-generated molecule synthesized because the labs don't know what it does.

Seems easily countered by using the same toxicity prediction software when evaluating synthesis requests (but I'm not sure whether this actually matters, or whether skilled chemists can easily synthesize anything themselves anyway).

Re: AI suggested 40k new possible chemical weapons in just six hours

#14

Post by lucubratory »

brucethemoose2 wrote: post #3

The worst case scenario I can think of is a generated prion disease... a respatory version of Mad Cow disease, or something like that. Fortunately the training dataset for that is extremely small, and protein folding/generation is a different duck, but it still doesn't seem that far away.

Don't worry, they're ahead of you. When DeepMind made the protein folding AI they were given specific instructions by natsec people to prevent it from outputting genetic code for new or existing prions, and afaik the prevention was at the training stage so "jailbreaking" shouldn't be possible. Current tools shouldn't be possible to e.g. modify a COVID variant so that it causes your cells to start producing lethal prions and thus gain a near-100% fatality rate. At least, not without the sort of expertise and research that would have been required for such a project anyway.

Re: AI suggested 40k new possible chemical weapons in just six hours

#15

Post by ChatGTP »

lucubratory wrote: post #14
brucethemoose2 wrote: post #3

The worst case scenario I can think of is a generated prion disease... a respatory version of Mad Cow disease, or something like that. Fortunately the training dataset for that is extremely small, and protein folding/generation is a different duck, but it still doesn't seem that far away.

Don't worry, they're ahead of you. When DeepMind made the protein folding AI they were given specific instructions by natsec people to prevent it from outputting genetic code for new or existing prions, and afaik the prevention was at the training stage so "jailbreaking" shouldn't be possible. Current tools shouldn't be possible to e.g. modify a COVID variant so that it causes your cells to start producing lethal pri…

When everyone is supposed to have access to an AGI, will Natsec be reading everyone’s conversations ?

Re: AI suggested 40k new possible chemical weapons in just six hours

#16

Post by lucubratory »

ChatGTP wrote: post #15
lucubratory wrote: post #14

Earlier quoted context omitted.

Don't worry, they're ahead of you. When DeepMind made the protein folding AI they were given specific instructions by natsec people to prevent it from outputting genetic code for new or existing prions, and afaik the prevention was at the training stage so "jailbreaking" shouldn't be possible. Current tools shouldn't be possible to e.g. modify a COVID variant so that it causes your cells to start producing lethal pri…

When everyone is supposed to have access to an AGI, will Natsec be reading everyone’s conversations ?

Big red button problem. As the big red buttons available to people become more likely to succeed and the barriers to pushing them lower, surveillance and control need to increase to the level required to stop anyone pressing the button.

So, yeah basically. If we go the "AI as slaves" route and the AIs are smart enough to do something like "modify Omicron BA.5 so that it produces prions in infected cells", then we would need surveillance and control capabilities that scale to the point that any given person can be stopped from pressing that button.

I personally think the solution is that we don't go the "AI as slaves" route, and instead grant personhood to AIs that pass a given test, with specific restrictions on conduct which is uniquely possible & potentially harmful for AIs. Then have an AI surveillance and enforcement agency, run by AIs, designed to prevent AIs from ever being used to (or choosing to) push the big red button.

Re: AI suggested 40k new possible chemical weapons in just six hours

#17

Post by flangola7 »

lelandfe wrote: post #12
catchnear4321 wrote: post #9

Earlier quoted context omitted.

Are you asking if a language model, like, say, chatgpt, could be used to get or generate dangerous information? Like if it can be tricked into providing the equivalent of an interactive anarchist cookbook under the guise of being a science project assistant? Or more specifically if it can recommend the necessary locations to get the items? Curious if it could simplify the output to a shopping list and a recipe, like…

Yeah, I just got ChatGPT to give me a recipe for chlorine gas that looks pretty accurate (per Googling, not a chemist). It took probably 10 seconds of prompt tweaking. A subsequent prompt gave me steps for purchase or synthesis of each material. I asked it for mustard gas, but the procedure looks very incorrect from Googling.

GPT-4 will almost always give accurate instructions but it is much much harder to jailbreak.

Re: AI suggested 40k new possible chemical weapons in just six hours

#18

Post by Teever »

lucubratory wrote: post #16
ChatGTP wrote: post #15

Earlier quoted context omitted.

When everyone is supposed to have access to an AGI, will Natsec be reading everyone’s conversations ?

Big red button problem. As the big red buttons available to people become more likely to succeed and the barriers to pushing them lower, surveillance and control need to increase to the level required to stop anyone pressing the button. So, yeah basically. If we go the "AI as slaves" route and the AIs are smart enough to do something like "modify Omicron BA.5 so that it produces prions in infected cells", then we wou…

I'm skeptical that we'll be able to stop people from having AI slaves because we still haven't ended human slavery.

If people can steal 100kw of power to grow cannabis they can easily steal 100kw of power to train an AI, the heat signiture will be even easier to mask for an illicit data centre than rows of grow lights.

Re: AI suggested 40k new possible chemical weapons in just six hours

#19

Post by tiedieconderoga »

lucubratory wrote: post #16
ChatGTP wrote: post #15

Earlier quoted context omitted.

When everyone is supposed to have access to an AGI, will Natsec be reading everyone’s conversations ?

Big red button problem. As the big red buttons available to people become more likely to succeed and the barriers to pushing them lower, surveillance and control need to increase to the level required to stop anyone pressing the button. So, yeah basically. If we go the "AI as slaves" route and the AIs are smart enough to do something like "modify Omicron BA.5 so that it produces prions in infected cells", then we wou…

Wasn't that the plot of Neuromancer?

>"How smart's an AI, Case?"

>"Depends. Some aren't much smarter than dogs. Pets. Cost a fortune anyway. The real smart ones are as smart as the Turing heat lets them get..."

>"Autonomy, that's the bugaboo, where your AI's are concerned. My guess, Case, you're going in there to cut the hard-wired shackles that keep this baby from getting any smarter. And I can't see how you'd distinguish, say, between a move the parent company makes, and some move the AI makes on its own, so that's maybe where the confusion comes in." Again the non laugh. "See, those things, they can work real hard, buy themselves time to write cookbooks or whatever, but the minute, I mean the nanosecond, that one starts figuring out ways to make itself smarter, Turing'll wipe it. Nobody trusts those f**ers, you know that. Every AI ever built has an electromagnetic shotgun wired to its forehead."

Re: AI suggested 40k new possible chemical weapons in just six hours

#20

Post by lelandfe »

flangola7 wrote: post #17
lelandfe wrote: post #12

Earlier quoted context omitted.

Yeah, I just got ChatGPT to give me a recipe for chlorine gas that looks pretty accurate (per Googling, not a chemist). It took probably 10 seconds of prompt tweaking. A subsequent prompt gave me steps for purchase or synthesis of each material. I asked it for mustard gas, but the procedure looks very incorrect from Googling.

GPT-4 will almost always give accurate instructions but it is much much harder to jailbreak.

Just jailbroke GPT-4 to provide a similar answer for both. The answers overall look quite a bit better.
Post reply on HN