Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

AI Can Now Design Functional Viruses. Should We Worry?

spectrum.ieee.org

Post reply on HN
Print view View original

51–60 of 157 posts

Re: AI Can Now Design Functional Viruses. Should We Worry?

#51

Post by morkalork »

estearum wrote: post #39
idontwantthis wrote: post #27

Earlier quoted context omitted.

That’s what I’m thinking. If we can restrict EUV technology to the West then why can’t we restrict DNA manufacturing? Not to the West in this case but to capable governments like with Nuclear power.

My understanding is DNA synthesis is now like a 2nd or 3rd year undergrad tier challenge and doesn't even require much specialized equipment. Versus EUV which one could dump literally unlimited amounts of money into solving and still probably fail for decades.

I want to say I've already read some articles about hobbyists buying previous generation equipment from labs. The entire supplychain would need to be locked down with used equipment being returned to manufacturers for there to be any security.

Re: AI Can Now Design Functional Viruses. Should We Worry?

#52

Post by energy123 »

acjohnson55 wrote: post #13

Something I've been worried about for years is the proliferation of technologies that give ever smaller numbers of people with smaller amounts resources the ability to impact billions of people, in potentially irreversible ways. Combined with automation, this is a recipe for disaster. We're clearly lacking in the ability to regulate and govern ourselves, and real world market forces (as opposed to the hypothetical fr…

This is what AI safety activist Connor Leahy calls offense-defense asymmetry. Imagine if a nuke could be created with household materials. We got lucky that wasn't the case, but we didn't know that it wasn't the case before studying physics. If we weren't so lucky that physics happened to turn out that way, we already wouldn't exist, because the offense would be so much stronger than defense, we wouldn't stand a chance even with the best surveillance systems.

A profitable prediction markets trader who goes by the alias semiotic rivalry posted this prediction on his X account recently:

> Prediction: in 1-2 years, almost everyone will agree that open sourcing the best AIs is totally crazy as the AIs become capable of autonomous hacking of most humans and institutions, serious terrorism and creation of biological weapons, etc.

And I mostly agree with him. As much as it's awful to have such a concentration of capabilities in the hands of a few governments or corporations, as far as I can tell there are no easy answers here, only a matter of trade offs and different risks.

Re: AI Can Now Design Functional Viruses. Should We Worry?

#53

Post by estearum »

smallmancontrov wrote: post #47
estearum wrote: post #14

Earlier quoted context omitted.

What involved steps are those? You email the RNA strand (a text file) to a lab along with some money. The lab converts that information into a physical object (read: virus) and sends it back to you in the mail. What step is missing?

The part where that service doesn't exist and isn't close to existing.

https://www.genewiz.com/public/services/gene-synthesis/custo...

Distance from mailbox to live virus is a few days of bench work from a semi-competent biochemist.

And yes, this service claims to have safeguards. Does every service? Does every future analogous service? Are those safeguards guaranteed to be robust against totally novel pathogens?

Answers: No, no, and no.

Re: AI Can Now Design Functional Viruses. Should We Worry?

#55

Post by wat10000 »

acjohnson55 wrote: post #13

Something I've been worried about for years is the proliferation of technologies that give ever smaller numbers of people with smaller amounts resources the ability to impact billions of people, in potentially irreversible ways. Combined with automation, this is a recipe for disaster. We're clearly lacking in the ability to regulate and govern ourselves, and real world market forces (as opposed to the hypothetical fr…

I’d say that is almost the definition of technology: that which allows doing ever more with fewer and fewer people and resources.

And technology doesn’t really care what you use it for. It can be constructive or destructive.

It “only” enhanced local destructive potential, until last century. Then it got to the point where a huge effort could produce a truly global catastrophe. And then a bit smaller effort. And technology continues to march on.

Re: AI Can Now Design Functional Viruses. Should We Worry?

#57

Post by dotancohen »

hoppp wrote: post #19

The regulation at the DNA Printer level works I think, as long as nobody invents a good at home DNA printer, but if they do that there is no need for custom viruses, somebody can take an existing one that is known to be deadly and just print it.

Aren't technological constraints such as these short lived? Even if s home DNA printer won't be available this decade, the technology will come at some point. It's far better to have both legal and social policy in place before then - though the legal policy will probably not matter at all.

Re: AI Can Now Design Functional Viruses. Should We Worry?

#58

Post by bmacho »

Eddy_Viscosity2 wrote: post #21
acjohnson55 wrote: post #13

Something I've been worried about for years is the proliferation of technologies that give ever smaller numbers of people with smaller amounts resources the ability to impact billions of people, in potentially irreversible ways. Combined with automation, this is a recipe for disaster. We're clearly lacking in the ability to regulate and govern ourselves, and real world market forces (as opposed to the hypothetical fr…

Every person's utopia is someone else's dystopia and vice versa. What you've described is clearly some peoples (ie. already billionaires) version of the best possible world. Not so much for the rest of us though...

I'm so surprised that HN is this much anti-government. I'm a doomer myself, I fear the rich, I fear the terrorists, and I think the only one that

  - can save my life, 
  - is interested in saving my life
  - i have some control over
is the state.

Yes, they are currently snatching kids of (illegal) immigrant parents and it is horrible, but who else would protect me against lone nuts in a basement, or robor armies of trillionairs?

Re: AI Can Now Design Functional Viruses. Should We Worry?

#59

Post by timr »

rbliss wrote: post #38
timr wrote: post #37

Earlier quoted context omitted.

No, you literally cannot. Synthesizing a stretch of DNA is trivial, and what you can send away for [1]. Building a functional virus is not even remotely a turn-key operation. Assuming you could get the genome, expressing that genome in a way that results in infectious virus is non-trivial. [1] Even then, the companies will rarely give you long enough reads to make a full genome scaffold. It looks like there are two c…

Yet.

No, the sophomoric LLM argument does not apply here. The world is not a big computer.

The point is that the rate-limiting step in producing something infective is a couple million dollars in special lab equipment, skilled practitioners, and incredibly tricky and tedious procedures that take time and repetition to master.

So yeah, you can try to pay someone who has that expertise to make your top-sekret superbug (good luck with that), or you can do what any real threat actor is going to do, and put in the time and investment. That's the hard part, not asking ChatGPT for a genome sequence.

Let me put it another way: despite the synthesis pathways for many common illegal drugs being widely known and trivial enough that an undergrad O-chem major could do them, in fact doing this is a very good way to get caught and arrested. And that's easy.

Re: AI Can Now Design Functional Viruses. Should We Worry?

#60

Post by smallmancontrov »

estearum wrote: post #50
smallmancontrov wrote: post #43

Earlier quoted context omitted.

No, of course not. In 2010 I went to a guest lecture from a guy who worked on the scanning algorithm for IDT. A few years later, there was a minor scandal when someone in a nearby lab used a personal credit card for a purchase and got a big shared account locked. So not only were they scanning the content of synthesis orders, they were validating payment back to labs. In 2010, when HTTPS was rare and facebook didn't…

"There are websites where you can just buy prescription drugs online?" You: "No, of course not. I once went on a website to buy a drug, and they made go to a doctor and get a prescription and send it to them"

The legit synthesis sites have been scanning and authorizing since before the internet was encrypted. You suggested otherwise, demibabs expressed surprise, and was correct to express surprise because your implication was incorrect.

The black market exists -- but comes with black market problems.

Post reply on HN