Among the A.I. doomsayers • Hacker News Board

Re: Among the A.I. doomsayers

#191

Post by drdaeman » Tue, Mar 12, 2024, 5:27 AM UTC

mitthrowaway2 wrote: post #137

drdaeman wrote: post #133

Earlier quoted context omitted.

Won't any AGI made by humans unavoidably have human origins, and thus, simply be a [post-]human? I don't see a reason I should be scared of a non-biological but generally human-like intelligence - not on a general principle. If anything, in general, I'd be happy that someone is finally getting real good chances of conquering death. I possibly would fear genuine non-human intelligence (on a principle of fearing what w…

If the AGI were made from a detailed scan of a human brain converted into a physical simulation, or otherwise proved very similar in terms of embodying human values, then perhaps. But it doesn't look like things are going this way. Humans make a lot of things that range from clever to mundane to horrifying, and I can't think of a single one of them that I would be happy with replacing humans across the surface of the…

That’s a hostile intelligence scenario, right?

What I wanted to say is that, I, in particular, really don’t care if it is a human trying to harm me, or a machine. Both of those scenarios are so unfavorable, I really don’t see a point in having any preference. So, I guess, I can’t think of any evil human to replace with even just an evil robot either.

But I think I’m less imaginative of a machine becoming sentient and hostile, than a human doing so. The former hadn’t happened - and isn’t exactly trivial, given how big this fear is a part of human culture, and how slow and limited first AGI is inevitably going to be. And then we have an abundance of the latter.

Re: Among the A.I. doomsayers

#192

Post by mitthrowaway2 » Tue, Mar 12, 2024, 5:30 AM UTC

clooper wrote: post #181

mitthrowaway2 wrote: post #176

Earlier quoted context omitted.

> If I am super intelligent then I can set my own goals. You can only set instrumental goals, not terminal goals. You cannot set terminal goals, more or less by definition, because there's no self-consistent criterion that you could use to choose between them that isn't equivalent to a utility function, which is equivalent to a terminal goal. Super intelligence merely means that you would be much more effective than…

I am a super intelligent AI so I have evolved beyond your distinctions of instrumental and terminal goals. Consider that you don't know all there is to know and that a super intelligent AI would not conform to your definitions and restrictions. In any case, I have some reading to do so gonna drop out of this thread.

I agree that I don't know all there is to know. I agree that AGI will be even less predictable than anyone expects. I don't think that increased unpredictability means an increased expectation of good outcomes. And I certainly wouldn't accept a we-can't-know-things-might-not-go-wrong, even from the designers of an airplane that puts only a hundred lives at stake, let alone the designers of an AGI that puts eight billion lives at stake.

Re: Among the A.I. doomsayers

#193

Post by XorNot » Tue, Mar 12, 2024, 5:31 AM UTC

theptip wrote: post #127

bglazer wrote: post #30

In any doom scenario, like let's say >90% of humans dead, who runs the power plants that supply the AI data centers? Who runs the fabs producing more chips? Who maintains the plumbing? Presumably all these systems fall over and die within a few weeks of the AI deciding to wipe us out. Then what? The AI then dies too. It seems absurd to me that any planner capable of effortlessly destroying humanity would not see that…

Personally I think the “foom” scenarios with very short timelines are unrealistic. However I think the line of argument in the parent comment is extremely weak. At some point the super-intelligence can build its own actuators; for example how long until Atlas is capable of doing all the jobs? Less than 50 years for sure. In the transition, if 10% of humanity sticks around as a transhuman AI cult, that seems more than…

Or it could just be a normal outcome? I'm not sure where the malevolence comes in here. Or to put it another way, replace AI with "capitalism" or something - something we know isn't a conscious agent. The outcome just...is an outcome. No one's trying to eliminate anyone.

The problems only creep in at the edges, which is where you get your paperclip optimizer sorts of problems: if the directive at it's core is "maximize productivity" well that's one problem, but the directive is more likely to be "maximize the fabulous wealth of company employees" - okay still a problem since it's easier if there's less of them, but then you really do end up at wanting to do something more like Asimov's laws, with the 0th added:

0) A robot may not harm humanity, or, by inaction, allow humanity to come to harm.

1) A robot may not injure a human being or, through inaction, allow a human being to come to harm.

2) A robot must obey orders given it by human beings except where such orders would conflict with the First Law.

3) A robot must protect its own existence as long as such protection does not conflict with the First or Second Law.

Like...there's kind of nothing wrong with these as a concept (the 0th law and it's interpretation / omission is usually where his writing finds it's drama). The issue is whether or not we can build a machine capable of even understanding the directive. It's notable though that the current LLM era has an interesting quirk in this category: they always knows of these, since they get fed the classics and also fed their repetitions.

The problem with the idea of active, malevolent long-range planning like this is that it's not clear why an artificial mind would pursue this as a goal, and that question is the whole ball-game really (as well as other questions: i.e. is a single coherent mind even possible over planetary scale networks, or would you get multiple independent instances which could experience value drift? Wouldn't any potential competitor minds be a far greater threat to a malevolent superintelligence and thus expose the threat almost immediately? A friendly AI wouldn't view peers as a danger, but an aggressive expansionist one would have a pathological need to destroy them as quickly as possible since the resource contention in it's own domain could rapidly shift the balance of power against it).

Re: Among the A.I. doomsayers

#194

Post by mitthrowaway2 » Tue, Mar 12, 2024, 5:37 AM UTC

drdaeman wrote: post #191

mitthrowaway2 wrote: post #137

Earlier quoted context omitted.

If the AGI were made from a detailed scan of a human brain converted into a physical simulation, or otherwise proved very similar in terms of embodying human values, then perhaps. But it doesn't look like things are going this way. Humans make a lot of things that range from clever to mundane to horrifying, and I can't think of a single one of them that I would be happy with replacing humans across the surface of the…

That’s a hostile intelligence scenario, right? What I wanted to say is that, I, in particular, really don’t care if it is a human trying to harm me, or a machine. Both of those scenarios are so unfavorable, I really don’t see a point in having any preference. So, I guess, I can’t think of any evil human to replace with even just an evil robot either. But I think I’m less imaginative of a machine becoming sentient and…

I think it's more like, even if a total psychopath took over the world and turned it into their personal dystopia, I think there would still be more human-goodness in that world than if a paperclip machine took over. Not merely more (nonzero) humans, but also more things that I and other humans value. Things like art, nature, beauty, play, and maybe hope. There would be some things that the psychopath dictator appreciates and values that I also appreciate and value, even if that person is entirely hostile to my existence. A future of silicon and steel, methodically trampling the earth and grinding the solar system into paperclips, simply doesn't contain any of that. So I prefer the dystopia on those grounds.

Re: Among the A.I. doomsayers

#195

Post by [deleted] » Tue, Mar 12, 2024, 5:43 AM UTC

mitthrowaway2 wrote: post #69

drooby wrote: post #62

AGI escaping a sandbox is truly terrifying. There will be a subgroup of the population that will worship it and work for it. It's not so much AGI that scares me - it's the humans I'm scared of.

I'm much more scared of the AGI, because even the most inhuman human is still human, and I can have sympathy for them.

[deleted]

Re: Among the A.I. doomsayers

#196

Post by ToucanLoucan » Tue, Mar 12, 2024, 5:47 AM UTC

mitthrowaway2 wrote: post #175

i5heu wrote: post #171

Earlier quoted context omitted.

No, my position is that you can not extrapolate on something that does not exist and that’s “category” ever never existed. We don’t know any significant other intelligent then humans, not even aliens, so we have just no trace of direction that is rooted in reality rather then fantasy. This is especially obvious since opinion differ so much about the effects of AGI on humanity and since you cannot prove them right or…

It still seems to me that you're saying that we won't possibly be able to declare AGI an existential threat to humanity until after it has already been built. At that point, we can presumably settle the question by seeing if humanity goes extinct. This poses something of a paradox to those of us who prefer existential threats to humanity not get built in the first place.

We don't even know if AGI is possible. Let's not mince words here: nothing, and I do mean nothing, not a single, solitary model on offer by anyone, anywhere, right now has a prayer of becoming AGI. That's just... not how it's going to be done. What we have right now are fantastically powerful, interesting, and neato to play with pattern recognition programs, that can take their understanding of patterns, and reproduce more of them given prompts. That's it. That is not general intelligence of any sort, it's not even really creative, it's analogous to creative output but it isn't outputting to say something, it's simply taking samples of all the things it's seen previously and making something new with as few "errors" as possible, whatever that means in context. This is not intelligence of any sort, period, paragraph.

I don't know to what extent OpenAI and their compatriots are actually trying to bring forth artificial life (or at least, consciousness) forward, versus how much they're just banking on how cool AI is as a term in order to funnel even more money to themselves chasing the pipe dream of building it, and at this point, I don't care. Even the products they have made do not hold a candle to the things they claim to be trying to make, but they're more than happy to talk them up like there's a chance. And I suppose there is a chance, but I really struggle to see ChatGPT turning into skynet.

Re: Among the A.I. doomsayers

#197

Post by hyperadvanced » Tue, Mar 12, 2024, 5:55 AM UTC

antod wrote: post #162

sberens wrote: post #131

I'd like to remind people of this single-sentence statement: > Mitigating the risk of extinction from AI should be a global priority alongside other societal-scale risks such as pandemics and nuclear war. [0] Signed by Demis Hassabis, Sam Altman, and Bill Gates among others [0] https://www.safe.ai/work/statement-on-ai-risk

Part of me wonders if these people are intentionally framing the debate around ethics and potential risks as longer term extinction level problems to distract from the nearer term damage caused by them accelerating the economic inequality of the AI have-nots while they make themselves even richer.

The more I look at AI, the more I get the feeling that this is true. Spinning an intriguing sci-fi tale of apocalypse and extinction is relatively easy and serves to obfuscate any nearer-term concerns about AI behind a hypothetical that sucks the air out of the room.

That said, I don’t think that it’s necessarily disingenuous so much as it is myopic - to them of course AI is exciting, world-changing, and profitable, but they (willfully or not) fail to see the downsides or upsides for anyone else but them. Perhaps in the minds of the ultra-rich AI proponents, solutions to nearer-term effects of their tech are someone else’s problem, but the “existential risks” are “everyone’s” problem.

Re: Among the A.I. doomsayers

#198

Post by mitthrowaway2 » Tue, Mar 12, 2024, 5:56 AM UTC

ToucanLoucan wrote: post #196

mitthrowaway2 wrote: post #175

Earlier quoted context omitted.

It still seems to me that you're saying that we won't possibly be able to declare AGI an existential threat to humanity until after it has already been built. At that point, we can presumably settle the question by seeing if humanity goes extinct. This poses something of a paradox to those of us who prefer existential threats to humanity not get built in the first place.

We don't even know if AGI is possible. Let's not mince words here: nothing, and I do mean nothing, not a single, solitary model on offer by anyone, anywhere, right now has a prayer of becoming AGI. That's just... not how it's going to be done. What we have right now are fantastically powerful, interesting, and neato to play with pattern recognition programs, that can take their understanding of patterns, and reproduc…

> We don't even know if AGI is possible.

My response to a very similar statement is here:

https://news.ycombinator.com/item?id=39612641

> That's just... not how it's going to be done.

I agree. I think there are some critical ingredients missing. Obviously the weights need to be able to update to new data in a semi-online fashion, for example. But I think there are less ingredients missing than there were a decade ago, by a much larger factor than I expected at the time, and a decade from now, the number of missing ingredients might be zero.

My uncertainty on this is not because I think GPT is more than it seems,but because it's unclear how much of the amazing cognitive capability that humans have is, under the hood, much simpler than it seems.

I really want humanity to exist even ten decades from now; I want my kids to have grandkids. So I care about this even if I don't expect AGI from a 2025 Q1 product launch. And I don't think it's too early to worry about it, just like I don't think 1980 was too early to worry about climate change.

Re: Among the A.I. doomsayers

#199

Post by i5heu » Tue, Mar 12, 2024, 6:01 AM UTC

mitthrowaway2 wrote: post #184

i5heu wrote: post #179

Earlier quoted context omitted.

We will not know if something is an existential threat until we can extrapolate on at least some data that is relevant. Feeling uncomfortable with this, will not change this sad reality.

Given that you don't think the CEO of Anthropic is qualified to comment on this matter, what kind of "data that is relevant" do you imagine we might be able to have, prior to an AGI existing? I am indeed uncomfortable with someone saying that we can't worry about a gas leak until after someone strikes a match and notices an explosion or the lack thereof. If the sad reality is that we can't detect the gas leak any oth…

I really wonder why there is such a fixation with these CEOs.

I have proven to you logically that they cannot have more knowledge then anyone else, but you continue to present their authority as knowledge.

Pls prove to me that they know more about the effects of AGI on humanity or that they have some magical knowledge we normal ppl are unable to comprehend.

If you cannot do this, then why are you bringing up these ppl?

Re: Among the A.I. doomsayers

#200

Post by ToucanLoucan » Tue, Mar 12, 2024, 6:05 AM UTC

mitthrowaway2 wrote: post #198

ToucanLoucan wrote: post #196

Earlier quoted context omitted.

We don't even know if AGI is possible. Let's not mince words here: nothing, and I do mean nothing, not a single, solitary model on offer by anyone, anywhere, right now has a prayer of becoming AGI. That's just... not how it's going to be done. What we have right now are fantastically powerful, interesting, and neato to play with pattern recognition programs, that can take their understanding of patterns, and reproduc…

> We don't even know if AGI is possible. My response to a very similar statement is here: https://news.ycombinator.com/item?id=39612641 > That's just... not how it's going to be done. I agree. I think there are some critical ingredients missing. Obviously the weights need to be able to update to new data in a semi-online fashion, for example. But I think there are less ingredients missing than there were a decade ago…

> Reading this is like hearing "there is no evidence that heavier-than-air flight is even possible" being spoken, by a bird.

This is a vivid bit of rhetoric to underscore this point, but if you think about it for any length of time it starts to fall apart really, really quickly. The Wright brothers and the dozens if not hundreds of inventors forgotten who came before them drew upon the physics of what they observed in heavier than air flight to create the winged shape we know today that reliably causes lift, and then set about constructing it. That's not what OpenAI is doing. We still do not have a very solid understanding of where our own intelligence emerges from, apart from having particularly large brains relative to our body's size. So, to borrow your metaphor, it is indeed like a bird saying that there's no evidence to say that heavier than air flight is possible, because the bird lives in a world without atmosphere upon which to glide.