Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Building an early warning system for LLM-aided biological threat creation

openai.com

Post reply on HN
Print view View original

171–180 of 190 posts

Re: Building an early warning system for LLM-aided biological threat creation

#171

Post by pushfoo »

visarga wrote: post #162
croniev wrote: post #115

Earlier quoted context omitted.

Would you say that GPT-4 can reason now? I am not convinced this is case, it seems like it has just become more consistent at providing us with an output that we consider reasonable because it was engineered precisely to do that.

> Would you say that GPT-4 can reason now? Let's assume reasoning entails going beyond the stochastic parrot level. Can LLMs have skills not demonstrated in the training set? Here is a paper demonstrating that GPT-4 can combine up to 5 skills from a set of 100, effectively covering 100^5 tuples of skills, while only seeing much fewer combinations in training on a specific topic. > simple probability calculations indi…

TL;DR: They're close enough to make people argue and publish papers about similarities to the human hippocampus

I have a hunch these models are approximating an important subset of what we call reasoning. In dangerously reductive terms, it's a question of how closely and how much of a function's output we can approximate.

There was at least one paper[1] showing similarities between AI models and the hippocampus. That lines up with another part of human neuroscience: at least part of human reasoning appears to take place inside the hippocampus itself [2].

From my neuroscience background, the takeaways seem to be:

* Carmack is right: we're missing some important bridging concepts for AGI.

* Whether current LLMs can reason depends on how you define reasoning

I'm unsure whether finding answers in those areas would be good thing. Instead of alignment issues or misuse, I'm more worried about how quickly people would overreact to it. We might already be seeing that in business.

1. https://arxiv.org/abs/2103.07356

2. https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3312239

Re: Building an early warning system for LLM-aided biological threat creation

#172

Post by foo3a9c4 »

jph00 wrote: post #168
foo3a9c4 wrote: post #154

Earlier quoted context omitted.

> There are far far more dollars available to people that are on the "AI Safety" bandwagon than to those pushing back against it. > The idea that the Upton Sinclair effect is the source of pushback against AI Safety zealotry, is getting things largely backwards AFAICT. > Folks that are stressing the importance of studying the impact of concentrated corporate power, or the risk of profit-driven AI deployment, and so f…

I'm not really trying to rebut Michael's argument -- I think it's true, to an extent, some of the time. But I think it's more true more of the time in the reverse direction. So I don't think it's a good argument. And more importantly, I think it fails to properly grapple with the ideas, instead using an ad hominem approach to discarding them somewhat thoughtless. On your last point, I do think it's important to note,…

(this discussion is quite nuanced so I apologize in advance for any uncharitable interpretations that I may make.)

> I'm not really trying to rebut Michael's argument -- I think it's true, to an extent, some of the time. But I think it's more true more of the time in the reverse direction.

I understand you to be saying:

Michael: Pro AI capabilities people are ignoring AIXR ideas because they are very excited about benefiting from (the funding of) future AI systems.

Reverse Direction: ainotkilleveryoneism people are ignoring AIXR ideas because they are very excited about benefiting from the funding of AI safety organizations.

And that (RD) is more frequently true than (M).

IMO both (RD) and (M) are true in many cases. IME it seems like (M) is true more often. But I haven't tried to gather any data and I wouldn't be surprised if it turned out to actually be the other way.

> So I don't think it's a good argument.

I might be misunderstanding you here because I don't see Michael making an argument at all. I just see him making the assertion (M).

> And more importantly, I think it fails to properly grapple with the ideas, instead using an ad hominem approach to discarding them somewhat thoughtless.

I am ambivalent toward this point. On one hand Michael is just making a straightforward (possibly false) empirical claim about the minds of certain people (specifically, a claim of the form: these people are doing X because of Y). It might really be the case that people are failing to grapple with AIXR ideas because they are so excited about benefiting from future AI tech, and if it were, then it seems like the sort of thing that it would be good to point out.

But OTOH he doesn't produce an argument against the claim "AIXR is just marketing hype." which is unfair to someone who has genuinely come to that conclusion via careful deliberation.

> On your last point, I do think it's important to note, and reflect carefully on, the extremely high overlap between those funding ai notkilleveryoneism and those funding capabilities development.

Thanks for pointing this out. Indeed, why are people who profess that AI has a not insignificant chance of killing everyone also starting companies that do AI capabilities development? Maybe they don't believe what they say and are just trying to get exclusive control of future AI technology. IMO there is a significant chance that some parties are doing just that. But even if that is true, then it might still be the case that ASI is an XR.

Re: Building an early warning system for LLM-aided biological threat creation

#173

Post by DougBTX »

habinero wrote: post #138
foo3a9c4 wrote: post #130

Earlier quoted context omitted.

> It’s definitely not the case. LLMs of any sort do not in any sense reason or understand anything. This seems like a claim about the way that the LLM neural net algorithm works. But AFAIK no one has a good understanding of how the LLM NNs work. Why are you so certain that the LLM NN isn't doing the reasoning-algorithm or the understanding-algorithm?

Neural networks are not new, and they're just mathematical systems. LLMs don't think. At all. They're basically glorified autocorrect. What they're good for is generating a lot of natural-sounding text that fools people into thinking there's more going on than there really is.

> and they're just mathematical systems

Obvious question: can Prolog do reasoning?

If your definition of reasoning excludes Prolog, then... I'm not sure what to say!

Re: Building an early warning system for LLM-aided biological threat creation

#175

Post by jefftk »

pushfoo wrote: post #139
jefftk wrote: post #121

Earlier quoted context omitted.

>> minimal control, i.e. checking DNA / RNA / protein sequences > They were already doing this 15 years ago. I work in this area. Some synthesis providers check, others don't. And the checking isn't great. (Some of my coworkers work on https://securedna.org which is trying to make this screening more robust.)

TL;DR: I don't understand how adding homomorphic encryption makes a cloud virus scanner for physical pathogens a better idea > Only authorized researchers should be able to obtain DNA permitting them to assemble pandemic-capable agents. Imagine this becomes legally mandatory. What happens when they get breached? For example: * Does it enable denial-of-service attacks by returning false positives hospitals legally can…

> I don't understand how adding homomorphic encryption makes a cloud virus scanner for physical pathogens a better idea

The reason to use a cloud service is that you can check whether someone is trying to synthesize something hazardous without distributing a list of the hazards. There are a lot of subtle ways to cause harm with biology, and you don't want to tell people where to look.

Then the reason to use homomorphic encryption is that otherwise the cloud service learns which sequences people are trying to synthesize. Biotech companies care a lot about keeping their in-progress work private, so they'd reject a cloud system without this.

> Does it enable denial-of-service attacks by returning false positives hospitals legally can't ignore?

Aside: I'm confused why you're saying "hospitals" -- medical providers don't synthesize things, it's researchers at universities and biotech companies. Maybe you're using "hospitals" as in "research hospitals" which is fine, but maybe you're using it because you're under the impression that synthesis is part of medical treatment?

To answer your question, though, if an attacker managed to add something normal and harmless to the DB then yes, a benchtop synthesizer would refuse to synthesize the sequence. The lab would escalate, it would be sorted out, there'd be a postmortem etc.

> Does it return false negatives as part of another attack? Example: printing known or novel virii which will be introduced in a specific lab?

Yes, if an attacker managed to remove a harmful sequence from the database then it wouldn't raise any flags if they or a confederate tried to synthesize that sequence.

Both of these cases come down to "it's important that you have good controls around what's in the DB, and a secure process for making changes."

> Metadata seems like it would enough to target the physical parts of an attack usefully, even without plaintext.

I'm not sure what you're saying here.

Re: Building an early warning system for LLM-aided biological threat creation

#176

Post by dTal »

nradov wrote: post #146
printerphobia wrote: post #66

>biological threat creation process (ideation, acquisition, magnification, formulation, and release) I remember watching a hacker/programmer who was livestreaming how to datamine the rna of the corona virus when covid first started. One of the crazy things he rambled about was how cheap it is for a layman to download a copy of it and synthesize it with some rna printing service. I haven't thought about that possibili…

LLMs change nothing. The sequences for various viruses are going to be published or leaked whether anyone involved uses LLMs or not. It's a total red herring.

A big value proposition of LLMs is their ability to synthesize and remix. Sure, you can access viral sequences today, but with an LLM you might be able to say "given these covid variants, make me a version as deadly as the first one and as virulent as the latest variant".

Re: Building an early warning system for LLM-aided biological threat creation

#177

Post by saint_fiasco »

kmeisthax wrote: post #126

Doesn't this completely deflate the selling point of AI? They force-fed a model the entire Internet and only got a statistically insignificant improvement over human performance.

Human performance costs minimum wage. AI products cost a lot to train but a lot less than a (first world) (educated) human's wage to run.

Re: Building an early warning system for LLM-aided biological threat creation

#178

Post by groby_b »

foo3a9c4 wrote: post #133
groby_b wrote: post #128

Earlier quoted context omitted.

We should also deeply worry about space aliens showing up and blasting us out of the sky. If they're sufficiently powerful, that could absolutely happen! Stop any radio emissions! xRisk is an absolutely stupid way to reason about AI. It's an unprovable risk that requires "mitigation just in case". All this is is saying "but if it were to happen, the cost is infinity, so any risk is a danger! Infinity times anything i…

Thanks for engaging in a discussion about AIXR. IMO it's important to figure out if we are actually about to kill ourselves or whether some people are just getting worked up over nothing. > We should also deeply worry about space aliens showing up and blasting us out of the sky. If they're sufficiently powerful, that could absolutely happen! Stop any radio emissions! If I believed that dangerous space aliens were lik…

Premise 3 is where the problem is, of course.

We have no idea how to build AGI. We know LLMs won't be it.

Alignment is a tool that works with LLMs, but we don't know if it will work for whatever produces AGI.

Even if we create AGI, we have no indication it is possible to build a orders-of-magnitude more "intelligent" thing. This is predicated entirely on the notion that if you can do it at scale, you get more, and there's no evidence thinking more makes for more intelligence.

Even if that were possible and we build an ASI, it's not at all clear this would lead to existential catastrophe. An ASI is presumably smart enough to see it's about to end the world as we know it, and knows where its power supply comes from.

This leaves us with an xrisk probability so close to zero it's virtually indistinguishable from zero. The only way to make it mean anything is "let's multiply it with infinity" - "it will end humanity, and my own survival is endangered".

Meanwhile, ordinary humans can use currently existing tools to end the world just fine. Nukes are readily available. We're obviously not really interested in public health. Climate refugees will be a giant problem soon-ish. The economy is very much a house of cards, but a house of cards that keeps society functioning as-is.

LLMs are a fantastic disinfo tool right now. There's a reasonably good chance they will calcify biases. They will cause large economic damage because 1) they lift up the baseline of work, and 2) they're just good enough that there's economic incentive to replace workers with it, but 3) they're shitty enough that the resulting output will ultimately be worse because we removed humans from the loop.

Those are actual risks. That we sweep under the carpet, because "xrisk" makes for much more grabby headlines.

Re: Building an early warning system for LLM-aided biological threat creation

#179

Post by groby_b »

hollerith wrote: post #135
groby_b wrote: post #128

Earlier quoted context omitted.

We should also deeply worry about space aliens showing up and blasting us out of the sky. If they're sufficiently powerful, that could absolutely happen! Stop any radio emissions! xRisk is an absolutely stupid way to reason about AI. It's an unprovable risk that requires "mitigation just in case". All this is is saying "but if it were to happen, the cost is infinity, so any risk is a danger! Infinity times anything i…

>the cost is infinity, so any risk is a danger That's not the argument. The argument is that human extinction is what you would naturally expect to happen if AI research continues on its present course unless you are biased because your income depends on AI research continuing unimpeded or you have an irrational emotional need to believe that technological progress is always good or you considered the question for 3…

"what you would naturally expect to happen "

Why? What is the reasoning this "is naturally expected"

"When sci-fi authors for example have treated the topic in fiction"

I'm sorry, but what you read in that book, saw in that movie isn't actually science. It's a cautionary tale about humans and what they are willing to do.

Re: Building an early warning system for LLM-aided biological threat creation

#180

Post by nradov »

dTal wrote: post #176
nradov wrote: post #146

Earlier quoted context omitted.

LLMs change nothing. The sequences for various viruses are going to be published or leaked whether anyone involved uses LLMs or not. It's a total red herring.

A big value proposition of LLMs is their ability to synthesize and remix. Sure, you can access viral sequences today, but with an LLM you might be able to say "given these covid variants, make me a version as deadly as the first one and as virulent as the latest variant".

That's not how viruses work.
Post reply on HN