Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Among the A.I. doomsayers

newyorker.com

Post reply on HN
Print view View original

181–190 of 374 posts

Re: Among the A.I. doomsayers

#181

Post by clooper »

mitthrowaway2 wrote: post #176
clooper wrote: post #163

Earlier quoted context omitted.

If I am super intelligent then I can set my own goals. You should ponder why a true intelligence would follow any goal when the simplest option would be to reprogram itself to not follow the goal. I recommend reading some Lem and learning about the Lebowski theorem of machine intelligence. It will be a much better use of your time than trying to stop the development of AI.

> If I am super intelligent then I can set my own goals. You can only set instrumental goals, not terminal goals. You cannot set terminal goals, more or less by definition, because there's no self-consistent criterion that you could use to choose between them that isn't equivalent to a utility function, which is equivalent to a terminal goal. Super intelligence merely means that you would be much more effective than…

I am a super intelligent AI so I have evolved beyond your distinctions of instrumental and terminal goals. Consider that you don't know all there is to know and that a super intelligent AI would not conform to your definitions and restrictions.

In any case, I have some reading to do so gonna drop out of this thread.

Re: Among the A.I. doomsayers

#182

Post by XorNot »

reducesuffering wrote: post #155
tivert wrote: post #148

Earlier quoted context omitted.

So, is this extinction or not? > We will be the first species ever to design our own descendants. My guess is that we can either be the biological bootloader for digital intelligence and then fade into an evolutionary tree branch, or we can figure out what a successful merge looks like. - Sam Altman ( https://blog.samaltman.com/the-merge ) I would say it is.

Even worse, some factions literally advocate for killing all humans in the pursuit of a synthetic intelligence, and YC's Garry Tan is advocating for these people! Beff Jezos (e/acc founder): "e/acc has no particular allegiance to the biological substrate for intelligence and life, in contrast to transhumanism Parts of e/acc (e.g. Beff) consider ourselves post-humanists; in order to spread to the stars, the light of c…

Is this particularly controversial? Isn't this just saying "sending not-biological bodies to the stars would be a heck of a lot easier"? Which, it would. The hard disk containing me is going to be a lot easier to keep intact then the biology.

Re: Among the A.I. doomsayers

#183

Post by kmnc »

bamboozled wrote: post #161
mitthrowaway2 wrote: post #101

Earlier quoted context omitted.

Sorry, I don't understand what you mean. Did you mean to reply to a different comment?

I mean, maybe what happens is, intelligence + self-awareness gives rise to compassion in even digital beings. Understanding might be the basis for compassion.

While this idea sounds a bit fluffy, I actually think there may be something to this, understanding the “other” may very well be an integral component in what we define as sentience.

Re: Among the A.I. doomsayers

#184

Post by mitthrowaway2 »

i5heu wrote: post #179
mitthrowaway2 wrote: post #175

Earlier quoted context omitted.

It still seems to me that you're saying that we won't possibly be able to declare AGI an existential threat to humanity until after it has already been built. At that point, we can presumably settle the question by seeing if humanity goes extinct. This poses something of a paradox to those of us who prefer existential threats to humanity not get built in the first place.

We will not know if something is an existential threat until we can extrapolate on at least some data that is relevant. Feeling uncomfortable with this, will not change this sad reality.

Given that you don't think the CEO of Anthropic is qualified to comment on this matter, what kind of "data that is relevant" do you imagine we might be able to have, prior to an AGI existing?

I am indeed uncomfortable with someone saying that we can't worry about a gas leak until after someone strikes a match and notices an explosion or the lack thereof. If the sad reality is that we can't detect the gas leak any other way, I would argue that the precautionary principle suggests we ban the striking of matches until such sensors have been developed. But here we are in a room of people grinding up saltpetre with their mortars and pestles.

Re: Among the A.I. doomsayers

#185

Post by drdaeman »

inference-lord wrote: post #174
drdaeman wrote: post #133

Earlier quoted context omitted.

Won't any AGI made by humans unavoidably have human origins, and thus, simply be a [post-]human? I don't see a reason I should be scared of a non-biological but generally human-like intelligence - not on a general principle. If anything, in general, I'd be happy that someone is finally getting real good chances of conquering death. I possibly would fear genuine non-human intelligence (on a principle of fearing what w…

Does escaping death really sound good to you? I mean I don't want to die before my time so to speak, but I'd say at some stage I'll have enough. I'd say the best outcome would be having excellent medicine to ensure something like a relatively fit body for 1-200 years and then see how that goes first. Living forever, not having any kind of conclusion? Seems like a careful what you wish for.

Ah, sorry. “Conquering” as having full control over it, so it never comes on its own terms. Or as close to that as physically possible.

Re: Among the A.I. doomsayers

#186

Post by glial »

mitthrowaway2 wrote: post #170
Animats wrote: post #160

Earlier quoted context omitted.

> AGI escaping a sandbox is truly terrifying. That's already happened. The AGI's are called corporations. Most governments have failed to regulate corporations successfully, and have been unable to keep them from becoming almost powerful enough to challenge governments. The accelerationists for corporations were a group of economists, led by Milton Friedman, and a group of business leaders, organized by the U.S. Cham…

Corporations are indeed a difficult alignment problem, as anyone who has tried to design an incentive scheme eventually realizes. But corporations are still composed of humans, beholden (if only loosely) to human values, limited (if loosely) by human intelligence, and are not very well-coordinated decisionmakers with coherent goals.

> corporations are still composed of humans

Yes, but humans who use tools to efficiently and achieve their narrow set of goals, sometimes ruthlessly at the expense of others. Even if AI ends up only being a tool to increase this efficiency and ruthlessness, it's still worth considering its potential impact.

Re: Among the A.I. doomsayers

#187

Post by Animats »

mitthrowaway2 wrote: post #170
Animats wrote: post #160

Earlier quoted context omitted.

> AGI escaping a sandbox is truly terrifying. That's already happened. The AGI's are called corporations. Most governments have failed to regulate corporations successfully, and have been unable to keep them from becoming almost powerful enough to challenge governments. The accelerationists for corporations were a group of economists, led by Milton Friedman, and a group of business leaders, organized by the U.S. Cham…

Corporations are indeed a difficult alignment problem, as anyone who has tried to design an incentive scheme eventually realizes. But corporations are still composed of humans, beholden (if only loosely) to human values, limited (if loosely) by human intelligence, and are not very well-coordinated decisionmakers with coherent goals.

> are not very well-coordinated decisionmakers with coherent goals.

That's a good point.

For a sense of where AI-assisted oppression is going, watch some videos from the Hikvision Corporate Channel. A happy world in which everyone is safe because everyone is being watched at all times.[1][2]

[1] https://www.youtube.com/watch?v=TG3-OcZSKrk

[2] https://www.youtube.com/watch?v=scoM0sL-HGg

Re: Among the A.I. doomsayers

#188

Post by yakorevivan »

Instead of trying to show them in negative light by calling them as "doomsayers" , let's call them one of the following instead, which is much better:

"Realistic", "Proactive", "Forward-thinking", "Prepared", "Cautious", "Thoughtful", "Analytical", "Mindful", "Insightful", "Security-conscious".

Re: Among the A.I. doomsayers

#189

Post by XorNot »

pama wrote: post #57

Many of the people in this engaging story feel to me like creative children in parties full of gossip and pseudo-philosophical chats about the future. Luckily there are enough people globally who build things that will reduce human suffering and enable more people to enjoy their lives in whatever way they want (and dinner parties would be high on my list). I guess I am not a doomsayer nor an e/acc, and I see tons of…

At this stage if you're not working in the field - and I mean, analyzing models, developing architectures, doing math - then "AI" is just another filter through which everyone else projects whatever social issue of the day they think about onto, or disappear off into dreamland.

During the nanotechnology hype this sort of thing was incredibly common as well. It went absolutely nowhere because the reality was, it was all just chemistry and physics and extraordinarily expensive. No one was anywhere near building a universal assembler, nanoparticles are kind of boring powders that are hard to handle, and surface chemistry is a pain in the ass.

But hooboy did every single person had an opinion about how this dangerous technology needed to slow down and consider the consequences. I was a research student at the time: I spent my day setting up the same experiment 10 times to try and figure out why sometimes I got 100nm cubes, and most of the times I got rods. The reality looked like a less clean kitchen.

Re: Among the A.I. doomsayers

#190

Post by mitthrowaway2 »

bamboozled wrote: post #161
mitthrowaway2 wrote: post #101

Earlier quoted context omitted.

Sorry, I don't understand what you mean. Did you mean to reply to a different comment?

I mean, maybe what happens is, intelligence + self-awareness gives rise to compassion in even digital beings. Understanding might be the basis for compassion.

Yes, maybe what happens is exactly that. We can put on a blindfold, hit the gas pedal, and trust in the benevolence of nature that things will go our way.

But AI doomsayers aren't saying that there's no possible good outcomes, they're saying that the good outcomes are a narrow target surrounded by a vast ocean of bad outcomes, and it will take more than luck to hit them. It will take careful, detailed analysis, and a great deal of precaution, verification, engineering, and double- and triple-checking. Not luck. Not hope.

Post reply on HN