Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Building an early warning system for LLM-aided biological threat creation

openai.com

Post reply on HN
Print view View original

131–140 of 190 posts

Re: Building an early warning system for LLM-aided biological threat creation

#131

Post by laughingman2 »

croniev wrote: post #115
apsec112 wrote: post #36

Way too many people extrapolate from "AI Foo can't do task X" to "AIs in general can never do task X", whether X is a good thing (like playing Go) or a bad thing (like helping build bioweapons). AI right now is pretty much the fastest-moving field in human history. GPT-4 can't revolutionize the economy, and it can't commit mass murder, but we simply don't know if GPT-7 can do either of those things (which likely won'…

Would you say that GPT-4 can reason now? I am not convinced this is case, it seems like it has just become more consistent at providing us with an output that we consider reasonable because it was engineered precisely to do that.

Research done by Prof. Subbarao on reasoning & planning with LLM seems to point in the negative.

https://www.youtube.com/watch?v=uTXXYi75QCU

He has lots of good threads distilling his research. https://twitter.com/rao2z

Re: Building an early warning system for LLM-aided biological threat creation

#132

Post by jph00 »

michael_nielsen wrote: post #129
digging wrote: post #48

Earlier quoted context omitted.

It's always really embarrassing to come to these comment sections and see a lot of smart people talk about how they're not being fooled by the "marketing hype" of existential AI risk. Literally the top of the page is saying that they have no conclusive evidence that ChatGPT could actually increase the risk of biological weapons. They are undertaking this effort because the question of how to stop any AI from ending h…

I'm not sure why they're so often so bad. I wonder if it's the Upton Sinclair effect; to paraphrase slightly: "It is difficult to get a person to understand something, when their hoped-for future wealth depends on not understanding it."

There are far far more dollars available to people that are on the "AI Safety" bandwagon than to those pushing back against it.

The idea that the Upton Sinclair effect is the source of pushback against AI Safety zealotry, is getting things largely backwards AFAICT.

Folks that are stressing the importance of studying the impact of concentrated corporate power, or the risk of profit-driven AI deployment, and so forth are receiving very little financial support.

Re: Building an early warning system for LLM-aided biological threat creation

#133

Post by foo3a9c4 »

groby_b wrote: post #128
digging wrote: post #48

Earlier quoted context omitted.

It's always really embarrassing to come to these comment sections and see a lot of smart people talk about how they're not being fooled by the "marketing hype" of existential AI risk. Literally the top of the page is saying that they have no conclusive evidence that ChatGPT could actually increase the risk of biological weapons. They are undertaking this effort because the question of how to stop any AI from ending h…

We should also deeply worry about space aliens showing up and blasting us out of the sky. If they're sufficiently powerful, that could absolutely happen! Stop any radio emissions! xRisk is an absolutely stupid way to reason about AI. It's an unprovable risk that requires "mitigation just in case". All this is is saying "but if it were to happen, the cost is infinity, so any risk is a danger! Infinity times anything i…

Thanks for engaging in a discussion about AIXR. IMO it's important to figure out if we are actually about to kill ourselves or whether some people are just getting worked up over nothing.

> We should also deeply worry about space aliens showing up and blasting us out of the sky. If they're sufficiently powerful, that could absolutely happen! Stop any radio emissions!

If I believed that dangerous space aliens were likely, then I would be interested in investigating ways to avert/survive such an encounter. This seems pretty rational to me, but maybe I'm confused.

> xRisk is an absolutely stupid way to reason about AI. It's an unprovable risk that requires "mitigation just in case".

By "unprovable risk" do you mean that it's literally impossible to know anything about the likelihood that dangerous algorithms could kill (nearly) all people on Earth?

> All this is is saying "but if it were to happen, the cost is infinity, so any risk is a danger! Infinity times anything is infinity!". It's playground reasoning.

Maybe you've seen people make that argument, but it strikes me as a strawman. Here is what I consider to be a better argument for not rushing ahead with capabilities development.

Premise 1. I value my own survival over just about anything else.

Premise 2. If an existential catastrophe occurs, then I will die.

Premise 3. If ASI is built before alignment is understood, then there is a significant chance of existential catastrophe.

Conclusion. So, I strongly prefer that ASI not be built until alignment is understood.

Re: Building an early warning system for LLM-aided biological threat creation

#134

Post by tehwebguy »

danielmarkbruce wrote: post #14
elzbardico wrote: post #4

Open AI is clearly overestimating the capabilities of its product. It is kind of funny actually.

Are they? Or is this marketing genius in several ways?

It's marketing, but it's ultimately intended for legislators.

Re: Building an early warning system for LLM-aided biological threat creation

#135

Post by hollerith »

groby_b wrote: post #128
digging wrote: post #48

Earlier quoted context omitted.

It's always really embarrassing to come to these comment sections and see a lot of smart people talk about how they're not being fooled by the "marketing hype" of existential AI risk. Literally the top of the page is saying that they have no conclusive evidence that ChatGPT could actually increase the risk of biological weapons. They are undertaking this effort because the question of how to stop any AI from ending h…

We should also deeply worry about space aliens showing up and blasting us out of the sky. If they're sufficiently powerful, that could absolutely happen! Stop any radio emissions! xRisk is an absolutely stupid way to reason about AI. It's an unprovable risk that requires "mitigation just in case". All this is is saying "but if it were to happen, the cost is infinity, so any risk is a danger! Infinity times anything i…

>the cost is infinity, so any risk is a danger

That's not the argument. The argument is that human extinction is what you would naturally expect to happen if AI research continues on its present course unless you are biased because your income depends on AI research continuing unimpeded or you have an irrational emotional need to believe that technological progress is always good or you considered the question for 3 minutes then held stubbornly to the conclusions of that 3 minutes of thinking.

When sci-fi authors for example have treated the topic in fiction (e.g., Vinge, Greg Bear, James Cameron's Terminator) most of the time the AI wipes out the species that created it.

Re: Building an early warning system for LLM-aided biological threat creation

#136

Post by Libcat99 »

marcellus23 wrote: post #112
zeofig wrote: post #103

Earlier quoted context omitted.

If physics allows it... a considerable assumption. In my view physics (or mathematics, or whatever) doesn't even allow an LLM to "reason", let alone manufacture novel bioweapons. The probability of Jesus returning is considerable too if we assume the bible is true.

What if you ignore LLMs specifically? I think that's the point that the GP was making. Do you believe it's mathematically impossible for any artificial machine to "reason"? Or just LLMs?

If humans can reason, then physics allows it, and software can too, eventually.

Re: Building an early warning system for LLM-aided biological threat creation

#137

Post by [deleted] »

foo3a9c4 wrote: post #130
Nifty3929 wrote: post #118

Earlier quoted context omitted.

It’s definitely not the case. LLMs of any sort do not in any sense reason or understand anything. They literally just make stuff up (technically just a continuation of whatever you fed in), which usually sounds good, is often true and sometimes even helpful. Because those are qualities of the training data that was used and is the basis for the stuff it’s making up.

> It’s definitely not the case. LLMs of any sort do not in any sense reason or understand anything. This seems like a claim about the way that the LLM neural net algorithm works. But AFAIK no one has a good understanding of how the LLM NNs work. Why are you so certain that the LLM NN isn't doing the reasoning-algorithm or the understanding-algorithm?

[deleted]

Re: Building an early warning system for LLM-aided biological threat creation

#138

Post by habinero »

foo3a9c4 wrote: post #130
Nifty3929 wrote: post #118

Earlier quoted context omitted.

It’s definitely not the case. LLMs of any sort do not in any sense reason or understand anything. They literally just make stuff up (technically just a continuation of whatever you fed in), which usually sounds good, is often true and sometimes even helpful. Because those are qualities of the training data that was used and is the basis for the stuff it’s making up.

> It’s definitely not the case. LLMs of any sort do not in any sense reason or understand anything. This seems like a claim about the way that the LLM neural net algorithm works. But AFAIK no one has a good understanding of how the LLM NNs work. Why are you so certain that the LLM NN isn't doing the reasoning-algorithm or the understanding-algorithm?

Neural networks are not new, and they're just mathematical systems.

LLMs don't think. At all. They're basically glorified autocorrect. What they're good for is generating a lot of natural-sounding text that fools people into thinking there's more going on than there really is.

Re: Building an early warning system for LLM-aided biological threat creation

#139

Post by pushfoo »

jefftk wrote: post #121
jdewerd wrote: post #111

Earlier quoted context omitted.

> minimal control, i.e. checking DNA / RNA / protein sequences They were already doing this 15 years ago. > It's frankly ridiculous there are no checks in place right now. It's frankly ridiculous that you think this.

>> minimal control, i.e. checking DNA / RNA / protein sequences > They were already doing this 15 years ago. I work in this area. Some synthesis providers check, others don't. And the checking isn't great. (Some of my coworkers work on https://securedna.org which is trying to make this screening more robust.)

TL;DR: I don't understand how adding homomorphic encryption makes a cloud virus scanner for physical pathogens a better idea

> Only authorized researchers should be able to obtain DNA permitting them to assemble pandemic-capable agents.

Imagine this becomes legally mandatory. What happens when they get breached? For example:

* Does it enable denial-of-service attacks by returning false positives hospitals legally can't ignore?

* Does it return false negatives as part of another attack? Example: printing known or novel virii which will be introduced in a specific lab?

* Does the attacker selectively perform these behaviors as part of an action against a specific person or area?

Metadata seems like it would enough to target the physical parts of an attack usefully, even without plaintext.

The best security & privacy approach I've seen in hospitals is keeping critical services on-prem, or at least strictly intranet-only. The latter ends up being forced by physical scale once a medical complex grows large enough to span multiple buildings.

The firmware embedding mentioned on their site seems better than cloud, yet still seems misguided. How will you debug problems, especially if you never have access to the plaintext? Formal methods and verified programs? Provably equivalent encrypted and plain operations?

Post reply on HN