Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

I'm sorry, you're not going to die from an AI-engineered supervirus

blog.genesmindsmachines.com

Post reply on HN
Print view View original

81–90 of 123 posts

Re: I'm sorry, you're not going to die from an AI-engineered supervirus

#81

Post by chrismartin »

areoform wrote: post #68
janalsncm wrote: post #27

> When people believe that AI could figure this out on its own, without first running extensive and expensive basic science experiments, they have left the realm of science and have entered what can only be described as religion. I strongly agree with this. Too many people have the Arthur C Clarke quote infecting their brains. So much of science is contingent on the physical world and cannot be determined from first…

I want to go further. A lot of people talking about AI bioweapons have no idea just how dangerous these weapons are. Or, rather they think they do, but unfortunately, there's a gulf in understanding between practitioners and commentators. I blame industrial illiteracy and post-literacy, but that's another discussion for another time. Anything that can kill other people can also kill you. I've posted this list before,…

This argument only matters in the case of agents who are faithfully taking direction from humans. An insufficiently-aligned autonomous agent (or perhaps a swarm of such agents) could have (or illicitly gain) access to robotic wet labs that are operable using something like Anthropic's Model Hardware Standard ( https://www.anthropic.com/news/model-hardware-standard-resea... ). Given this ability to manipulate the real-life laboratory, the agents could cook something terrible without any human in charge. This is not different in principle from the agent swarm that successfully attacked Hugging Face, despite receiving no human direction to do so.

Also, your own real-life examples show how "Anything that can kill other people can also kill you" has failed to stop people from tinkering with dangerous viruses! I'm not sure if you intend for these examples to be reassuring, but I'm not reassured.

> A lot of the people talking about bioweapons of doom are saying that standard graduate level knowledge is an existential risk to humanity.

The stronger version of the argument is: an insufficiently-aligned AI (which, being non-biological, is immune to viruses) having standard graduate level bio knowledge _plus_ the ability to operate a wet lab via tool calls (coming very soon if not here already!) adds up to existential risk to humanity.

You don't even need a maximally-automated lab if humans are willing to tele-operate equipment at the agent's direction, and those humans don't need to fully understand what's going on.

Re: I'm sorry, you're not going to die from an AI-engineered supervirus

#82

Post by mikestylz »

crazygringo wrote: post #52
mikestylz wrote: post #26

The magical level of intelligence required is not really a flaw of the argument's deductions, it is its very premise. If you believe that there is a possibility of an AI working on itself, you get the approximate differential equation d/dt f(t) >= C*f(t). In plain English, at least some proportion of development's progress goes back into future development. The (least case growth, assuming equality) solution to this…

> What we really need for such a statement is concrete physical evidence that a virus is simply incapable of having such facilities. A major focus of the article is about the fundamental tradeoffs in how viruses operate. Was that not enough for you?

It is not, because we should not expect a manufactured technology to behave exactly as an evolved one. My understanding is there is not much incentive for a virus to kill off all its hosts. When they do it is largely an accident, and it is not uncommon for a severe virus to eventually get replaced by a less virulent and thus more successful strain. It's pretty hard to hill-climb evolution to such a point since the virus could starve itself in the process. And induction on evolution feels misguided to me. Consider our species. A million years ago it would have seemed completely reasonable to state that the absence of advanced tool use in the kingdom of life suggests it is biologically impossible. After all, such a thing had not occurred in hundreds of millions of years!

To be clear, I am not saying that such a virus will exist eventually. I am just noting that the arguments displayed here are not completely tight, and there are unknown unknowns at play. Some things are unfalsifiable. This is common.

Re: I'm sorry, you're not going to die from an AI-engineered supervirus

#83

Post by alevskaya »

naasking wrote: post #65
alevskaya wrote: post #41

Earlier quoted context omitted.

I was a biophysicist and experimental biologist for decades, I designed viral vectors and cell therapies, etc. What protects us here isn't our ignorance vs an "ASI". It's the fact that the complexity of molecular physics scales exponentially in particle number. No amount of thinking is going to punch through that. You've got to do experiments. The slow loop through reality is not optional, and in the case of "supervi…

We're already doing those experiments. Tailor made mRNA vaccines targeted to one's own specific cancer mutations can be bought right now. The techniques are getting more sophisticated and more targeted every year. Our ability to predict what happens at these levels is improving by leaps and bounds too thanks to AI like AlphaFold. Anyone can be reasonably confident that no AI or teen can do this now or in the next few…

I worked in immuno-oncology: cancer vaccines work in melanoma where many things work because of the neoantigen abundance, they've generally been very mixed in efficacy. And even BionTech's BNT111 failed in melanoma! We have hope for these approaches but the reality of this stuff is way more nuanced than you think it is.

Alphafold can't reliably predict thermal energy landscapes or make functional predictions - and how could it? It wasn't trained on anything that could capture structure - function relationships.

Again, most people just have no idea how hard - fundamentally hard - molecular physics is to predict, and how necessary experiments are for any development of biological systems.

Re: I'm sorry, you're not going to die from an AI-engineered supervirus

#85

Post by areoform »

thaumasiotes wrote: post #77
areoform wrote: post #68

Earlier quoted context omitted.

I want to go further. A lot of people talking about AI bioweapons have no idea just how dangerous these weapons are. Or, rather they think they do, but unfortunately, there's a gulf in understanding between practitioners and commentators. I blame industrial illiteracy and post-literacy, but that's another discussion for another time. Anything that can kill other people can also kill you. I've posted this list before,…

> It's worth repeating again, > Anything lethal enough to kill other humans is lethal enough to kill you. It's also completely false; you carry tons of diseases that are capable of killing isolated islanders and incapable of hurting anyone connected to the general run of humanity.

I often attend and host talks by researchers working on space exploration, and one finding that's been interesting is immune dysregulation in isolation posing risks to astronauts and explorers. Here's a review on the topic, https://www.frontiersin.org/journals/immunology/articles/10....

This type of dysregulation also exists in AIDs. For the edge case of the islanders, then the level of risk and lethality our germs pose for isolated islanders with modern medical care is somewhat disputed, but it would be deeply unethical to ever test. We just don't know. So I won't pretend to know here.

However, these are edge cases that don't matter, because bioweapons, i.e. things deliberately designed to kill, just aren't the same as a dozen isolated islanders, astronauts on a mission to mars, or someone who is severely immunocompromised.

Weapons are designed to kill healthy people. The doomsdays being proposed are super-bioweapons that kill billions of humans. I think it's safe to say that as a rule of thumb, by and large, anything that lethal is lethal enough to kill you.

Re: I'm sorry, you're not going to die from an AI-engineered supervirus

#86

Post by alevskaya »

georgemcbay wrote: post #73
jsnell wrote: post #47

Native americans had like a 95% population mortality rate from infectious diseases in the first generation of European settlement, no? Measles, smallpox, flu, etc. The mortality rate from those diseases was higher than today because the population had no immunity at all, leading to both higher transmission rates and higher mortality rates. This would be true of an engineered virus as well. And it wasn't just the one…

> So I don't find the bulk of the article about it being impossible for a virus to have this effect all that compelling. I think the other flaw the article has it that it assumes the bad actor is doing targeted design. That they or the AI has some specific virus in mind meeting a set of predefined criteria and they are fighting the biology to hit that predefined target. But what if the AI-enabled bad actor is a true…

[dead]

Re: I'm sorry, you're not going to die from an AI-engineered supervirus

#87

Post by thaumasiotes »

fergal_reid wrote: post #46

I am not a biology person, but, candidly, this is a bad article, because the author is generalizing from the properties of viruses that tend to arise naturally, and using this to argue about the properties of deliberately designed viruses. Firstly, it's not even clear that naturally occuring viruses can't be extremely lethal. There are examples (eg black death). Even the existence of rare examples should be concernin…

> Firstly, it's not even clear that naturally occuring viruses can't be extremely lethal. There are examples (eg black death).

You realize the black death didn't arrive and then go away again, right?

It was extremely lethal, once.

Re: I'm sorry, you're not going to die from an AI-engineered supervirus

#88

Post by busyant »

colingauvin wrote: post #43

Oh my God THANK YOU. I do not have the words how tired I am of hearing tech people and economists talk about biology.

As a biologist whose PhD research was on the genetics of microbial pathogens, I am far more concerned than the author.

I think he trots out a lot of straw-man arguments (e.g., teenager making the weapon, "super"virus, etc.).

Those scenarios are purposely far-fetched so that he can ridicule them and thereby leave the impression that all the AI/bioweapon concerns are similarly far-fetched.

He spends a lot of "tokens" discussing AI-generated viruses. But bioweapons can be much simpler than a virus--they can be small molecules that inactivate essential molecular targets in humans. This is something that was discussed at length when I was in grad school and it wasn't dismissed as hyperbolic hokum.

So, is AI going to develop a supervirus? Probably not. But it doesn't need to.

Instead, I speculate that AI can "help" in making bioweapons ...

a. more potent

b. less detectable

c. easier to disseminate

You still need humans in the lab to run the experiments, but I think it's foolish to dismiss the worries about AI in such a condescending way.

Re: I'm sorry, you're not going to die from an AI-engineered supervirus

#89

Post by alevskaya »

woopwoop wrote: post #50

The author seems to be arguing that it is biologically impossible or something to have a virus with the same transmission properties as COVID but higher virulence. I don't understand what principle he is invoking to make this claim.

Because fatality and virulence stand in fundamental tension with each other when it comes to epidemiology. You can't magic that tension away or just invent other incompatible properties of a hypothetical virus (stealth, latency, virulence, fatality, etc etc etc) just because you want to tell a ghost story. Biology: (immunology, cell biology, virology) impose constraints!

Re: I'm sorry, you're not going to die from an AI-engineered supervirus

#90

Post by areoform »

chrismartin wrote: post #81
areoform wrote: post #68

Earlier quoted context omitted.

I want to go further. A lot of people talking about AI bioweapons have no idea just how dangerous these weapons are. Or, rather they think they do, but unfortunately, there's a gulf in understanding between practitioners and commentators. I blame industrial illiteracy and post-literacy, but that's another discussion for another time. Anything that can kill other people can also kill you. I've posted this list before,…

This argument only matters in the case of agents who are faithfully taking direction from humans. An insufficiently-aligned autonomous agent (or perhaps a swarm of such agents) could have (or illicitly gain) access to robotic wet labs that are operable using something like Anthropic's Model Hardware Standard ( https://www.anthropic.com/news/model-hardware-standard-resea... ). Given this ability to manipulate the real…

> access to robotic wet labs that are tool-callable

How? It's worth asking the question and doing the experiment, have you ever tried making a bioweapon?

For most of us, that's not going to be case, so the next best thing is to read accounts and reports of bioweapon programs. These things are... messy. And expensive.

As a very bad analogy, this is the biology equivalent of saying that I can make a nuke because I bought a drill, a screwdriver and a $200 centrifuge.

Or, saying that I can make Google because I opened MacOS' terminal.

Some people have jobs that are dangerous and necessary to save lives. Others have jobs that are dangerous and involve killing other people.

The real risk will emerge from malcontents in these jobs and positions.

For most of my life, for all its faults, the US Military has been one of the most responsible and competent organizations in the world when it comes to WMDs (and even they lose nukes see the many broken arrow incidents).

But despite some of the most thorough security clearance processes in the world, the only successful terror attack involving bioweapons on US soil was done by a disgruntled / insane researcher at Fort Detrick. https://en.wikipedia.org/wiki/2001_anthrax_attacks

That's the real risk, and let's be very honest here, do you think Anthropic and OpenAI are going to put a filter on the LLM they sell to the US Military?

The other case is Aum, and they fucked up deployment, btw (again making a bioweapon is harder than you'd think). And as I have detailed elsewhere, Anthropic and OpenAI would have sold them licenses too! https://news.ycombinator.com/item?id=49092899

They will say they will have audits in place, but the US Military had those in 2001.

Post reply on HN