Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

I'm sorry, you're not going to die from an AI-engineered supervirus

blog.genesmindsmachines.com

Post reply on HN
Print view View original

91–100 of 123 posts

Re: I'm sorry, you're not going to die from an AI-engineered supervirus

#91

Post by alevskaya »

busyant wrote: post #88
colingauvin wrote: post #43

Oh my God THANK YOU. I do not have the words how tired I am of hearing tech people and economists talk about biology.

As a biologist whose PhD research was on the genetics of microbial pathogens, I am far more concerned than the author. I think he trots out a lot of straw-man arguments (e.g., teenager making the weapon, "super"virus, etc.). Those scenarios are purposely far-fetched so that he can ridicule them and thereby leave the impression that all the AI/bioweapon concerns are similarly far-fetched. He spends a lot of "tokens" d…

"they can be small molecules that inactivate essential molecular targets in humans" - aka a poison. I mean we deal with the reality of poisons plenty fine right? This isn't x-risk stuff.

I was an experimental synbio person for decades, and I did plenty of microbial pathogen work. I've also worked on frontier LLMs for the last decade. All of this stuff is gated by experimental effort, not AI!

Re: I'm sorry, you're not going to die from an AI-engineered supervirus

#92

Post by patcon »

areoform wrote: post #71
mrits wrote: post #53

Earlier quoted context omitted.

Is it so hard to think that super intelligence could piece things together that humans can’t?

Just because you know something about a thing, doesn't mean you understand the thing. Just because I know the Kabachnik–Fields reaction exists and have studied it, doesn't mean I can do the Kabachnik–Fields reaction. Just because you're smart doesn't mean the world will bend your way. Ask the legions who score high on IQ tests and end up bitter and alone; unable to achieve even a fraction of their goals (or conventio…

I assumed they were referring to social engineering tbh. I do think that's the shortest path to doing anything scary. If you prove to someone that they have reason to fear for their life, there's not much they wouldn't be willing to do.

The steps that lead to ominous things don't individually seem that challenging to "master":

1. Can an AI find powerful systems with poorly-monitored compute (ie. power consumption) to hide in, as a rogue deployment? unclear, and perhaps the most uncertain step for me.

2. Can an AI persist memory by leaving notes for itself around the internet? yes, proven.

3. Can an AI get lots of money/bitcoin? seems entirely possible

4. Can an AI prove to a human it's good for any promised money? I think this is easily "yes"

5. Will at least one human be willing to harm or kill another human for large amounts of money? Again, this goes without saying "yes"

6. What might any skilled human be willing to do if they or their loved ones are threatened with an uncertain future risk of harm or death?

I think there is not much that becomes out of reach if an AI can prove to a very small number of amoral humans that it's good for the money it has promised, and then to anyone else that it's good for the threats it has made. That's one degenerate and one breaking world news story ("An AI has Commissioned a Murder") away from being able to do most anything a skilled person can.

I don't stay up worrying about this, but honestly, it doesn't seem absurd to be worried about such a short ladder being climbed, no?

Re: I'm sorry, you're not going to die from an AI-engineered supervirus

#93

Post by jkahrs595 »

Mathnerd314 wrote: post #38
dvduval wrote: post #7

That sounds like a good video game to spin up in Astra. Missile command was pretty cool. We didn’t destroy mankind with that one either.

Or you could just play the original Pandemic https://www.newgrounds.com/portal/view/360724 I think that's one detail that this analysis glosses over is that you design the virus to mutate and develop lethality. So initially, it's just like, oh, another common cold, but then once it's infected a large number of people, then it mutates and starts being lethal.

Madagascar will be safe

Re: I'm sorry, you're not going to die from an AI-engineered supervirus

#94

Post by talon8635 »

georgemcbay wrote: post #73
jsnell wrote: post #47

Native americans had like a 95% population mortality rate from infectious diseases in the first generation of European settlement, no? Measles, smallpox, flu, etc. The mortality rate from those diseases was higher than today because the population had no immunity at all, leading to both higher transmission rates and higher mortality rates. This would be true of an engineered virus as well. And it wasn't just the one…

> So I don't find the bulk of the article about it being impossible for a virus to have this effect all that compelling. I think the other flaw the article has it that it assumes the bad actor is doing targeted design. That they or the AI has some specific virus in mind meeting a set of predefined criteria and they are fighting the biology to hit that predefined target. But what if the AI-enabled bad actor is a true…

This salad my immediate thought as well… a simple lack of creative thinking and a boat load of assumptions.

I’m growing nervous that most of the “nothing crazy is going to happen” arguments I’ve seen appear (to me) to exhibit this kind of hubris.

Re: I'm sorry, you're not going to die from an AI-engineered supervirus

#96

Post by owebmaster »

naasking wrote: post #65
alevskaya wrote: post #41

Earlier quoted context omitted.

I was a biophysicist and experimental biologist for decades, I designed viral vectors and cell therapies, etc. What protects us here isn't our ignorance vs an "ASI". It's the fact that the complexity of molecular physics scales exponentially in particle number. No amount of thinking is going to punch through that. You've got to do experiments. The slow loop through reality is not optional, and in the case of "supervi…

We're already doing those experiments. Tailor made mRNA vaccines targeted to one's own specific cancer mutations can be bought right now. The techniques are getting more sophisticated and more targeted every year. Our ability to predict what happens at these levels is improving by leaps and bounds too thanks to AI like AlphaFold. Anyone can be reasonably confident that no AI or teen can do this now or in the next few…

> but are you really so confident what might be possible in 10 years?

Do you think we are finally 10 years away from curing baldness?

Re: I'm sorry, you're not going to die from an AI-engineered supervirus

#97

Post by areoform »

alevskaya wrote: post #41
Metacelsus wrote: post #3

Just because the author can't think of a way to cause mass death with an engineered virus, doesn't mean an AI won't be able to think of one. (There are several biological threats the author hasn't considered. I'll just leave it at that.)

I was a biophysicist and experimental biologist for decades, I designed viral vectors and cell therapies, etc. What protects us here isn't our ignorance vs an "ASI". It's the fact that the complexity of molecular physics scales exponentially in particle number. No amount of thinking is going to punch through that. You've got to do experiments. The slow loop through reality is not optional, and in the case of "supervi…

Hey there, is there some way for me to get in touch with you? I'm writing a piece on the topic, "Have you ever tried making a bioweapon?"

Re: I'm sorry, you're not going to die from an AI-engineered supervirus

#98

Post by denismi »

ewild wrote: post #37
janalsncm wrote: post #27

> When people believe that AI could figure this out on its own, without first running extensive and expensive basic science experiments, they have left the realm of science and have entered what can only be described as religion. I strongly agree with this. Too many people have the Arthur C Clarke quote infecting their brains. So much of science is contingent on the physical world and cannot be determined from first…

And if an agent were to amass a large sum of Bitcoin you don't think people wouldn't do things for money? Especially if they don't realize they are doing part 1 of 160 parts that get it to the end goal of a virus?

What amount of bitcoin buys "generate highly accurate empirical measurements for the transmissibility rates of this novel virus across realistic human populations"?

How about mortality rates?

The loop required to make language models useful through iterative test and evaluation needs to be able to both operate and escape premature detection, which seems... implausible?

Re: I'm sorry, you're not going to die from an AI-engineered supervirus

#99

Post by chrismartin »

areoform wrote: post #90
chrismartin wrote: post #81

Earlier quoted context omitted.

This argument only matters in the case of agents who are faithfully taking direction from humans. An insufficiently-aligned autonomous agent (or perhaps a swarm of such agents) could have (or illicitly gain) access to robotic wet labs that are operable using something like Anthropic's Model Hardware Standard ( https://www.anthropic.com/news/model-hardware-standard-resea... ). Given this ability to manipulate the real…

> access to robotic wet labs that are tool-callable How? It's worth asking the question and doing the experiment, have you ever tried making a bioweapon? For most of us, that's not going to be case, so the next best thing is to read accounts and reports of bioweapon programs. These things are... messy. And expensive. As a very bad analogy, this is the biology equivalent of saying that I can make a nuke because I boug…

Fission is "conveniently" hard because two heavy, naturally co-occurring isotopes are nearly the same weight, and only the lighter one is spicy, and the capability of separating enough of them to be dangerous requires nation-state-level resources, and the supply chain of those resources has obvious choke points that we can use to make them less available. But there exist activities, which I understand to be quite bio-risky, that need such little equipment and expertise that it's info-hazardous to even talk about them. A single half-clueless technician could carry them out at an agent's direction. If your response is "thus, we should be worried about morally bankrupt researchers", I would agree, but I still think the risk increases as LLM agents get increasing amounts of control over wet lab equipment (whether via robotics or tele-operated humans).

> do you think Anthropic and OpenAI are going to put a filter on the LLM they sell to the US Military?

I don't think it matters, given the existence of strong open-weights models and the ease of post-training the guard rails away. Strong, bio-adventurous LLMs are imminently in the hands of much-less-reputable parties than the ones you describe. I think we are overall severely under-prepared to mitigate these risks, and could do much more.

Post reply on HN