Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Among the A.I. doomsayers

newyorker.com

Post reply on HN
Print view View original

121–130 of 374 posts

Re: Among the A.I. doomsayers

#121

Post by mitthrowaway2 »

bamboozled wrote: post #72
drooby wrote: post #62

AGI escaping a sandbox is truly terrifying. There will be a subgroup of the population that will worship it and work for it. It's not so much AGI that scares me - it's the humans I'm scared of.

Isn’t ChatGPT et all already a general intelligence that’s escaped the sandbox ?

It hasn't escaped the sandbox because it hasn't figured out how to walk, but the dev teams have raced to open the door as widely as possible and unbolt the hinges. It's kind of funny to look back at the early 2000s when people were saying "of course, they'll know to unplug it when it asks for an internet connection or tries to execute arbitrary code"; in reality the first and second things we did were add an API for internet access and a python plugin.

Re: Among the A.I. doomsayers

#122

Post by TillE »

nsainsbury wrote: post #98

We're not going to make it to anywhere close to AGI before we see widespread and systematic societal and environmental collapse on almost all fronts due to climate change. We're in the middle of the sixth mass extinction right now ( https://en.wikipedia.org/wiki/Holocene_extinction ), we're in unparalleled territory with ocean warming: ( https://climatereanalyzer.org/clim/sst_daily/ ) and as a society we are utterly…

People really seem to enjoy the evil/powerful AGI scenario though. It's definitely a fun one. I would recommend reading "Metamorphosis of Prime Intellect", having a nice time with that, then coming back to the real world.

Re: Among the A.I. doomsayers

#123

Post by danielmarkbruce »

clooper wrote: post #78
gcr wrote: post #75

Earlier quoted context omitted.

I think this point is underrated, but I’d press further. How do we know this has not already happened? Agencies do not need to be born of gradient descent and certainly do not need to look like matrix multiplications to be deemed “artificially intelligent.” Religion is one such system for example, as are cults, addictions, brands, war strategies, organizing campaigns, governments, or (borrowing from chaos magick) any…

Many institutions are collective/artificial intelligence because as you said these institutions have objectives and take actions to impose their will on the world in order to achieve their goals.

"intelligence" is too strong a word for it. It's "a will of it's own" perhaps... but at least in my experience that will is a loooonggg way from intelligent.

Re: Among the A.I. doomsayers

#124

Post by _cs2017_ »

jodrellblank wrote: post #18
codelord wrote: post #12

We live in a world where proven maniacs (e.g. Putin) have access to an arsenal of nuclear weapons that can essentially make the earth uninhabitable for all humans. That's a very real possibility (with no ifs and buts and maybes) that exists now and we have learned to live with it. Yet somehow the hypothetical scenario of a human exterminator super-intelligent AI is getting all the coverage.

Making the Earth uninhabitable to all humans is bad news for Putin, a human who lives on Earth. He derives his power from hundreds of millions of people in Russia working, and his lifestyle from hundreds of millions of people around the world buying Russian exports so an attack on everyone is also going to harm him. AI wouldn’t be human, it wouldn’t necessarily object to a world uninhabitable to humans, or be hurt by…

Innumerable times quite reasonable political leaders have started wars that resulted in their own deaths; and sometimes resulted in the destruction of civilization as they knew it (most recently - World War I, which effectively put an end to the European monarchies).

This suggests that giving nukes to reasonable political leaders presents a high and concrete risk. Giving nukes to political leaders with crazy ideas presents an even greater risk.

Climate change seems to be another high and concrete risk.

AI escaping and taking over the world can be also considered a risk, but it's far more remote. It is similar to the risk of aliens attacking earth after detecting its TV and radio transmissions. Some people might argue that we should slow down the work on AI. Others may demand we also ban all TVs and radios on earth. Both proposals seem to be a bit of an overkill given how remote the associated risks are. Especially given that we have the far more pressing risks to address.

The concerns about AI risks are like a drunk driver going 90 mph on a country road and suddenly deciding to stop saying "goddamn" just in case Jesus returns and punishes those who invoked God's name in vain in violation of the Third Commandment.

Re: Among the A.I. doomsayers

#126

Post by HankB99 »

clooper wrote: post #74
HankB99 wrote: post #49

If I let my imagination run wild, I can imagine that at some point AI becomes in some way sentient. By that I mean it gains reasoning, some sort of understanding, and motivation. I wonder what the possibility is that this AI will decide that a pitched battle is going to waste resources and risk humans pulling the plug. What if it understood that and instead operated so subtly that it was not obvious it was controllin…

Lem wrote about this in Summa Technologiae. He called the field Intellectronics and explored what would happen once machine/electronic intelligence surpassed human intelligence: https://en.m.wikipedia.org/wiki/Summa_Technologiae

Not surprising that I'm not the first to think of this.

Re: Among the A.I. doomsayers

#127

Post by theptip »

bglazer wrote: post #30

In any doom scenario, like let's say >90% of humans dead, who runs the power plants that supply the AI data centers? Who runs the fabs producing more chips? Who maintains the plumbing? Presumably all these systems fall over and die within a few weeks of the AI deciding to wipe us out. Then what? The AI then dies too. It seems absurd to me that any planner capable of effortlessly destroying humanity would not see that…

Personally I think the “foom” scenarios with very short timelines are unrealistic.

However I think the line of argument in the parent comment is extremely weak. At some point the super-intelligence can build its own actuators; for example how long until Atlas is capable of doing all the jobs? Less than 50 years for sure.

In the transition, if 10% of humanity sticks around as a transhuman AI cult, that seems more than enough to bridge to nanotech, robots, or whatever is sufficient to replace humans.

If you prefer to think in more stepwise fashion, imagine BigCo creates an ASI. Every employee is fabulously wealthy, and over the course of a generation the company expands to take over the entire global economy. At no point is there a bad deal for any humans. Life might look great for a while. This would also be the obvious strategy for any super-intelligent entity that cannot yet survive without humans; you just need to minimize the chances of major wars and conflicts until your robots come online.

A digital mind would have no problem planning and executing a strategy measured in centuries, if need be.

The foom scenarios require extreme discontinuities in intelligence; an entity smart enough to synthesize nanomachines from existing tech and without doing physical experiments, or other IMO improbable outcomes.

But there are plenty of terrible non-foom outcomes.

Re: Among the A.I. doomsayers

#128

Post by woopsn »

jjeaff wrote: post #16
dkjaudyeqooe wrote: post #3

People have been watching too many movies. War Games, Terminator, it's not like we haven't been forewarned of the dangers. Yet somehow we're going to hand over power to AI such that it destroys us. Or somehow the AI is going to be extremely malign, determined to overcome and destroy and will outsmart us. Somehow we won't notice, even after repeated, melodramatic reminders, and won't neuter the ability of AI to act ou…

I am not super-intelligent, but I myself can imagine lots of ways that a super intelligence might outsmart an off switch. One being to hide what it is doing until it is able to write and distribute a virus to the outside world that distributes itself. Or perhaps it could convince some other humans to physically take over and stop anyone from flipping the off switch. The question you should ask yourself is how much da…

How much damage could one do with ~20 terawatts of brain power? A lot. But nothing clandestine.

Re: Among the A.I. doomsayers

#129

Post by mitthrowaway2 »

clooper wrote: post #83
thom wrote: post #51

Earlier quoted context omitted.

There are extremely high capability entities (people, companies, governments) that aren’t comically evil on the surface but nevertheless immiserate large groups of humans. Not all of them, not all of the time, but not none of them. What is your plan that no AI ever gains enough power to harm significant numbers of people either on purpose or by accident, just once? What safeguards do you envision that can’t be ignore…

Controlling people with violence and fear are evolutionary adaptations for ultra violent social primates. AI has no such pressures imposed on it so there is no reason to expect true AI will have the same drives as ultra violent predators like humans.

Keep following that line of thought, don't stop there! Why did those violent behaviors succeed in being selected for? Why didn't pacifism get selected for instead? Why does the government maintain a monopoly on violence, and why is it that the nth-degree consequence for violating societal laws is violence, instead of a monetary fine or a sternly-worded letter? Is there perhaps some kind of game-theoretic equilibrium that causes agents who employ violence -- even as a measure of last resort, after peaceful negotiation -- to succeed in achieving their objectives?

People, corporations, organizations, and nation states who employ violence do so when and because they expect it to work. An AI would need to be very specifically designed not to do that; it wouldn't be the default output of a search for a successful strategy.

Re: Among the A.I. doomsayers

#130

Post by HankB99 »

p1esk wrote: post #84
HankB99 wrote: post #49

If I let my imagination run wild, I can imagine that at some point AI becomes in some way sentient. By that I mean it gains reasoning, some sort of understanding, and motivation. I wonder what the possibility is that this AI will decide that a pitched battle is going to waste resources and risk humans pulling the plug. What if it understood that and instead operated so subtly that it was not obvious it was controllin…

If AI does ever become sentient, it will probably reason and decide about eliminating large swaths of human population in a similar fashion as we reason and decide about eliminating large swaths of ant population.

Ants? That's how aliens might view us. A super AI might find value in minions.
Post reply on HN