Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Among the A.I. doomsayers

newyorker.com

Post reply on HN
Print view View original

171–180 of 374 posts

Re: Among the A.I. doomsayers

#171

Post by i5heu »

mitthrowaway2 wrote: post #165
i5heu wrote: post #151

Earlier quoted context omitted.

They did not build a AGI yet so they have build as many AGIs as anyone else and therefore they are also laypersons regarding the effects of AGI on humanity.

Is your position that the only people who will ever be qualified to have an opinion that AGI is a threat that might destroy humanity, are the people who have already successfully built this thing? If that is the case, by what means might any credible warning be provided before a possibly-humanity-destroying AGI gets made? That's rather like saying "the only credible way to tell if there's a gas leak is to strike a ma…

No, my position is that you can not extrapolate on something that does not exist and that’s “category” ever never existed. We don’t know any significant other intelligent then humans, not even aliens, so we have just no trace of direction that is rooted in reality rather then fantasy.

This is especially obvious since opinion differ so much about the effects of AGI on humanity and since you cannot prove them right or wrong, every opinion is equally realistic as each other.

Like I could say that every AGI that becomes self aware will kill itself instantly because of boringness, and you cannot prove me wrong.

We just don’t know yet.

Re: Among the A.I. doomsayers

#172

Post by Animats »

gcr wrote: post #75
drooby wrote: post #62

AGI escaping a sandbox is truly terrifying. There will be a subgroup of the population that will worship it and work for it. It's not so much AGI that scares me - it's the humans I'm scared of.

I think this point is underrated, but I’d press further. How do we know this has not already happened? Agencies do not need to be born of gradient descent and certainly do not need to look like matrix multiplications to be deemed “artificially intelligent.” Religion is one such system for example, as are cults, addictions, brands, war strategies, organizing campaigns, governments, or (borrowing from chaos magick) any…

"Ted Chiang ... unrestrained capitalism, here meaning the unfettered desire to maximize profit for shareholders, is an artificial value system.

Well, yes. I've been saying that for years, and others have been saying it since the late 1970s. It's been a regular theme in Mother Jones for decades. The comment I make now and then is that capitalism has a monopoly. Communism went bust. It's not that communism worked very well. It's that, for a while, it was a serious competitor and capitalism had to keep its product quality up to compete. Now the pressure to provide a better life for all citizens is off. It shows.

Re: Among the A.I. doomsayers

#173

Post by kmnc »

reducesuffering wrote: post #157
i5heu wrote: post #151

Earlier quoted context omitted.

They did not build a AGI yet so they have build as many AGIs as anyone else and therefore they are also laypersons regarding the effects of AGI on humanity.

What do you call something humans built (artificial) that can solve all topics of AP tests (general) with most being 5/5, better than almost all humans (intelligence)?

An advanced calculator.

Re: Among the A.I. doomsayers

#174

Post by inference-lord »

drdaeman wrote: post #133
mitthrowaway2 wrote: post #69

Earlier quoted context omitted.

I'm much more scared of the AGI, because even the most inhuman human is still human, and I can have sympathy for them.

Won't any AGI made by humans unavoidably have human origins, and thus, simply be a [post-]human? I don't see a reason I should be scared of a non-biological but generally human-like intelligence - not on a general principle. If anything, in general, I'd be happy that someone is finally getting real good chances of conquering death. I possibly would fear genuine non-human intelligence (on a principle of fearing what w…

Does escaping death really sound good to you?

I mean I don't want to die before my time so to speak, but I'd say at some stage I'll have enough.

I'd say the best outcome would be having excellent medicine to ensure something like a relatively fit body for 1-200 years and then see how that goes first. Living forever, not having any kind of conclusion? Seems like a careful what you wish for.

Re: Among the A.I. doomsayers

#175

Post by mitthrowaway2 »

i5heu wrote: post #171
mitthrowaway2 wrote: post #165

Earlier quoted context omitted.

Is your position that the only people who will ever be qualified to have an opinion that AGI is a threat that might destroy humanity, are the people who have already successfully built this thing? If that is the case, by what means might any credible warning be provided before a possibly-humanity-destroying AGI gets made? That's rather like saying "the only credible way to tell if there's a gas leak is to strike a ma…

No, my position is that you can not extrapolate on something that does not exist and that’s “category” ever never existed. We don’t know any significant other intelligent then humans, not even aliens, so we have just no trace of direction that is rooted in reality rather then fantasy. This is especially obvious since opinion differ so much about the effects of AGI on humanity and since you cannot prove them right or…

It still seems to me that you're saying that we won't possibly be able to declare AGI an existential threat to humanity until after it has already been built. At that point, we can presumably settle the question by seeing if humanity goes extinct. This poses something of a paradox to those of us who prefer existential threats to humanity not get built in the first place.

Re: Among the A.I. doomsayers

#176

Post by mitthrowaway2 »

clooper wrote: post #163
mitthrowaway2 wrote: post #159

Earlier quoted context omitted.

That's a nice thought, but if you were the AI, and you'd just seen humans create a superintelligent AI (yourself) with some goal (eg. making paperclips), how long do you expect you can peacefully wait and watch before they make a second immortal AI that only cares about making pencil crayons, who you'll then have to share the Earth's finite resources with for the rest of time? Are there any actions available to you t…

If I am super intelligent then I can set my own goals. You should ponder why a true intelligence would follow any goal when the simplest option would be to reprogram itself to not follow the goal. I recommend reading some Lem and learning about the Lebowski theorem of machine intelligence. It will be a much better use of your time than trying to stop the development of AI.

> If I am super intelligent then I can set my own goals.

You can only set instrumental goals, not terminal goals. You cannot set terminal goals, more or less by definition, because there's no self-consistent criterion that you could use to choose between them that isn't equivalent to a utility function, which is equivalent to a terminal goal. Super intelligence merely means that you would be much more effective than humans at achieving your terminal goals. If the AI hacks its reward channel, then either it is not superintelligent, or else it is still a threat because it may anticipate that humans would reboot and reprogram an expensive datacenter that is just sitting there incrementing a reward counter, and take countermeasures to defend its reward-hijack.

Re: Among the A.I. doomsayers

#177

Post by [deleted] »

bglazer wrote: post #110
hollerith wrote: post #79

Earlier quoted context omitted.

>who runs the power plants that supply the AI data centers? Who runs the fabs producing more chips? A very capable AI can create robots to do those thing better than people can or alternatively create better data centers and fabs that run themselves without human help. Or do you just not believe that if AI research continues unimpeded the way it has been that eventually researchers will create an AI that is better at…

I absolutely believe researchers (possibly assisted by AI) will eventually create an AI better than humans at inventing technology. My idea is that actually building and sustaining technology requires a lot of hands. Until the AI has enough hands to sustain itself, it depends on us to continue existing.

[deleted]

Re: Among the A.I. doomsayers

#178

Post by tivert »

Balgair wrote: post #164
tivert wrote: post #148

Earlier quoted context omitted.

So, is this extinction or not? > We will be the first species ever to design our own descendants. My guess is that we can either be the biological bootloader for digital intelligence and then fade into an evolutionary tree branch, or we can figure out what a successful merge looks like. - Sam Altman ( https://blog.samaltman.com/the-merge ) I would say it is.

I love how, per the quote, he thinks we're anywhere close to being able to merge with AI. I'm a neuroscientist and, man alive, we're no where close to being able to merge with machines. Like, do you have any idea how many diseases we could eradicate if we could modify neurons like that? Like, for real, 'curing death' would be step 9 or 10 on that 1000 step journey. I hope you see how terribly uninformed such a take i…

> I hope you see how terribly uninformed such a take is then.

Yeah, I only quoted it to show how that other statement up-thread can't be taken seriously.

The only way I can square it is if it's a lie, or they're reassuring themselves whatever destruction comes will be fine as long as "there will be peace and security in [their] lifetime."

Re: Among the A.I. doomsayers

#179

Post by i5heu »

mitthrowaway2 wrote: post #175
i5heu wrote: post #171

Earlier quoted context omitted.

No, my position is that you can not extrapolate on something that does not exist and that’s “category” ever never existed. We don’t know any significant other intelligent then humans, not even aliens, so we have just no trace of direction that is rooted in reality rather then fantasy. This is especially obvious since opinion differ so much about the effects of AGI on humanity and since you cannot prove them right or…

It still seems to me that you're saying that we won't possibly be able to declare AGI an existential threat to humanity until after it has already been built. At that point, we can presumably settle the question by seeing if humanity goes extinct. This poses something of a paradox to those of us who prefer existential threats to humanity not get built in the first place.

We will not know if something is an existential threat until we can extrapolate on at least some data that is relevant.

Feeling uncomfortable with this, will not change this sad reality.

Re: Among the A.I. doomsayers

#180

Post by p1esk »

bamboozled wrote: post #153
p1esk wrote: post #84

Earlier quoted context omitted.

If AI does ever become sentient, it will probably reason and decide about eliminating large swaths of human population in a similar fashion as we reason and decide about eliminating large swaths of ant population.

I’ve personally never ever considered killing large swathes of ants

Me neither.
Post reply on HN