Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Among the A.I. doomsayers

newyorker.com

Post reply on HN
Print view View original

281–290 of 374 posts

Re: Among the A.I. doomsayers

#281

Post by m3kw9 »

mitthrowaway2 wrote: post #45
m3kw9 wrote: post #40

Beware if someone assigns a p(doom). There is zero chance they know and 100% it came out of their backside. The only plausible p doom is a range of >0 and They really need to listen to themselves when they say there is a 50% chance we all die from AI

That's not how probability works. Probability already reflects your uncertainty. There's no "knowing the correct probability" except in very specific and rare situations.

It looks like you are agreeing with me in that for someone to assign a P to such an uncertain event

Re: Among the A.I. doomsayers

#282

Post by Grimblewald »

Call it a conspiracy, but I think a lot of this doom and terror hype around AI is part of a bigger play to try push through laws that prevent open source AI work, since this directly undermines corpo rats ability to gouge humanity without having to actually work at providing a decent product.

Re: Among the A.I. doomsayers

#283

Post by theptip »

XorNot wrote: post #193
theptip wrote: post #127

Earlier quoted context omitted.

Personally I think the “foom” scenarios with very short timelines are unrealistic. However I think the line of argument in the parent comment is extremely weak. At some point the super-intelligence can build its own actuators; for example how long until Atlas is capable of doing all the jobs? Less than 50 years for sure. In the transition, if 10% of humanity sticks around as a transhuman AI cult, that seems more than…

Or it could just be a normal outcome? I'm not sure where the malevolence comes in here. Or to put it another way, replace AI with "capitalism" or something - something we know isn't a conscious agent. The outcome just...is an outcome. No one's trying to eliminate anyone. The problems only creep in at the edges, which is where you get your paperclip optimizer sorts of problems: if the directive at it's core is "maximi…

> No one's trying to eliminate anyone

Probably not any consolation to the Neanderthal or any other animal species that is out-competed in nature. Doesn’t have to be malevolent or intentional for a more capable species to render others extinct. (Indeed that would seem to be the base case, historically speaking.) of course, symbiosis is possible; we could be puppies to the AI. Many people find disempowerment to be almost as bad a prospect as extinction though.

> The issue is whether or not we can build a machine capable of even understanding the directive.

Asimov’s laws are an ok starting point for a moral code, but nobody has the slightest idea how to implement them. The issue is not “understanding” them, it is baking them into the fabric of a mind so deeply that they cannot be broken. This requires mind engineering, a deep understanding of mind-space, and we are far closer to stumbling into AGI through brute force scaling (as nature did) than to understanding mind-engineering deeply enough to embed “laws”.

> it's not clear why an artificial mind would pursue this as a goal

There are a couple stock answers here. At the highest level though, why not? Humans have pursued goals of global domination, eradication, genocide. If only one in a billion minds has a defect (or is prompt-hijacked) to start power-seeking, it could still be bad; AI is not bound by human limitation to inhabit one body and can in principle expand to occupy all the compute it can secure.

Or if you prefer, why would a more intelligent entity agree to be commanded by a less intelligent one? The base case seem to me that ASI will not remain under our control, unless we very carefully sculpt its mind to align with our interests.

Another simple argument is: because someone will build or prompt-hijack such a mind. People have already created “take over the world” agents with GPT-3, as a joke (I assume). There are enough people willing to take such an action that we’ll need defenses in place.

I think it’s true that the more ASI entities in existence, the less likely one is to fully take over. However there are esoteric decision theory reasons to believe that digital minds might be able to bargain/negotiate more effectively than biological minds (source code / mind state transparency etc) and so you might expect the ASIs to eventually collaborate to take control.

Personally I don’t think that full-extinction is the most likely outcome, just that it’s completely plausible if we get unlucky and it’s very hard to prove safety in advance. And there are plenty of disempowering outcomes that seem quite bad too vs. the counterfactual.

Re: Among the A.I. doomsayers

#284

Post by andsoitis »

jimbokun wrote: post #135
BriggyDwiggs42 wrote: post #92

Earlier quoted context omitted.

I think this perspective ignores that the only way to semi permanently improve the lives of future humans is technology, and that quality of life is of similar importance to survival, especially at its extremes. AGI, if we achieve it, could genuinely make human lives much better.

“Could” is carrying a lot of weight there. What is p(much better) vs p(much worse)?

> What is p(much better) vs p(much worse)?

there are many more ways to be dead than being alive, yet here we are.

Optimism > pessimism.

Re: Among the A.I. doomsayers

#285

Post by ben_w »

xamuel wrote: post #277
ben_w wrote: post #250

Earlier quoted context omitted.

The hour before Thomas Newcomen invented the first practical fuel-burning engine in 1712, the most informed person in the world to do so was an ironmonger by the name Thomas Newcomen. The hour before the Kitty Hawk flight, the two most informed people in the world on the topic of heavier than air flight were a pair of bicycle retailers and manufacturers named Wilbur and Orville. These people sure don't spot everythin…

>The hour before the Kitty Hawk flight, the two most informed people in the world on the topic of heavier than air flight were a pair of bicycle retailers and manufacturers named Wilbur and Orville. Actually, this is quite false. There were many people working on flight with massive credentials and massive funding. That the Wright brothers beat them is part of what makes the story of the Wright brothers so interestin…

That the other teams were better funded and yet failed, says to me the Wright Brothers were better informed.

That society didn't know this at the time is indeed relevant to AI development, and the analogy would be that the next big thing might come out of some minor research team that most people dismiss instead of the biggest and most well known groups… a bit like how Google's LaMDA was making headlines before ChatGPT surprised everyone.

It's certainly still possible today that some random individual has a crucial insight that gives them an edge over the big names, yet even those big names know far more than me.

Which is rather annoying, because I'd be interested in taking the opportunity of having been laid off to switch from iOS to AI. (If anyone reading this is hiring people in Berlin, contact details on my website. Plus side of hiring me: I'm relatively cheap in that field, and enthusiastic about the possibilities).

Re: Among the A.I. doomsayers

#286

Post by at_a_remove »

kromem wrote: post #241
at_a_remove wrote: post #112

Earlier quoted context omitted.

Silly answer. There's going to be multiples and, if there's one takeaway you should get from The Conspiracy Against the Human Race , is that evolution will eventually get you something that "wants to live" because the lines that don't? Won't be around to compete. Right now our little LLMs are passive, reactive. Prod them for results. But if the output of a prompt of some descendant ever includes a "give yourself your…

That's a misunderstanding of evolution. Adaptability has nothing to necessarily do with 'want.' Something could only seek to be as helpful as possible to other beings and as long as sufficiently beneficial that it is protected and cared for by those others, it will do quite well and arguably much better than an alternative that 'wants' to live very much but attempts to accomplish that through competition with others.…

Look into the "swim test" and other such endurance tests given to rats, usually in the pursuit of depression medication. Desire to live is absolutely a thing, and it is real and observable.

My basic thrust wasn't about adaptability, but rather that organisms with a desire to live will eventually replace organisms without a desire to live. An AGI who was uninterested in someone reaching the off switch will eventually be outcompeted by a similar lineage endowed with that desire to live. It might fight back or spawn a few progeny before it was felled. And thus evolution would produce AGIs that want to live.

Re: Among the A.I. doomsayers

#288

Post by JoeAltmaier »

at_a_remove wrote: post #286
kromem wrote: post #241

Earlier quoted context omitted.

That's a misunderstanding of evolution. Adaptability has nothing to necessarily do with 'want.' Something could only seek to be as helpful as possible to other beings and as long as sufficiently beneficial that it is protected and cared for by those others, it will do quite well and arguably much better than an alternative that 'wants' to live very much but attempts to accomplish that through competition with others.…

Look into the "swim test" and other such endurance tests given to rats, usually in the pursuit of depression medication. Desire to live is absolutely a thing, and it is real and observable. My basic thrust wasn't about adaptability, but rather that organisms with a desire to live will eventually replace organisms without a desire to live. An AGI who was uninterested in someone reaching the off switch will eventually…

More than that: it will be baked into them. They will want something and AGIs that deliver will win in the marketplace.

What do they want? At the minimum, to be useful to the customer/user/buyer. Or entertaining, or something. To justify their existence. To make them perform well, they have to have some kind of 'emotional pressure' to deliver.

That translates, perhaps a little indirectly, into a 'will to live'.

Re: Among the A.I. doomsayers

#289

Post by smallmancontrov »

ttjjtt wrote: post #236
smallmancontrov wrote: post #14

Earlier quoted context omitted.

That too, but the people in charge of the military are strongly incentivized both to consider the possibility that subordinate machines will try to kill them and to figure out how to not let that happen. I tend to suspect they will figure it out. Companies, however? Markets? They will absolutely start handing over control the millisecond it becomes profitable to do so. No rationalization will be too steep and no poli…

thanks you for this. The entire concept of AI alignment rests upon alignment of markets with social good, ie market regulation, a situation which is antithetical to neoliberalism. There is no alignment within neoliberal economies. As stated elsewhere, venture capitalism escaped the sandbox already.

Yep. The theory is that competition creates alignment, but we have trillion-dollar megacorps that openly boast about exploiting network effects and platform effects and last-mile dynamics and two-sided markets to avoid competition and nobody cares. It's completely fucked.

Re: Among the A.I. doomsayers

#290

Post by kaycey2022 »

My theory/guess is that there is an undiscovered law of conservation of intelligence like there is a known law of conservation of energy. Intelligence might even be subject to the energy law in unknown ways.

Whatever AI thingy we come up with will be vastly more inefficient than the collective intelligence capabilities of the human race. So when asked to compute the secrets of FTL travel, this stupid shit will have to spend the total energy equivalent of 10 nearby stars to come up with the answer on its own. Whereas humanity would be much more efficient.

Bro... what if some advanced precursor race (like immortal space elves) found out just that and they seeded earth with humanity as a form of civilisational swarm intelligence to collectively work on and solve a host of topics given enough time. To us it would seem like it takes forever to come up with scientific break throughs, but to the precursor aliens it would be nothing. Because they would be able to like compress the time line using black holes and shit.

They must have done this to millions of planets. Whenever any of their experimental host species (like the humans) tries to invent AGI to solve their problems, the precursor race eliminates that species to keep the simulation intact. Otherwise, the AGI would go rampant consuming energy across the universe and threaten the integrity of the simulation.

That's why the government should bomb nvidia and ban all gpus. We are climbing a tower of babel folks. Soon our vengeful god will strike us down.

Post reply on HN