Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Gradual Disempowerment: How Even Incremental AI Progress Poses Existential Risks

arxiv.org

Post reply on HN
Print view View original

71–80 of 87 posts

Re: Gradual Disempowerment: How Even Incremental AI Progress Poses Existential Risks

#71

Post by cyrillite »

Jordan-117 wrote: post #63

This just underscores the feeling that most of the problems people have with AI are actually problems with rampant capitalism. Negative externalities, regulatory capture, the tragedy of the commons -- AI merely intensifies them. I've heard it said that corporations are in many ways the forerunners of the singularity, able to act with superhuman effectiveness and marshall resources on a world-altering scale in pursuit…

I’d push it further than you, even, and have been thinking this way for a while. The economy is AI. I know that’s a ridiculously simplistic way to put it and a network of individual actors doesn’t function identically, but for all intents and purposes we’re massive amounts of distributed compute running a “capitalist” algorithm and it isn’t perfectly aligned with us either. We don’t have AI problems, we have a generic class of algorithm problems that keep popping up wherever agents interact dynamically.

Re: Gradual Disempowerment: How Even Incremental AI Progress Poses Existential Risks

#72

Post by alephnerd »

pram wrote: post #66
alephnerd wrote: post #50

Earlier quoted context omitted.

Naw, was before my time. I did see Terminator though. Did Wargames age well? I tried watching a couple iconic 80s movies like Scarface but I just couldn't stand it.

It's OK, but the climax is absolutely classic. "A strange game. The only winning move is not to play."

Sounds intriguing. I know broad strokes about the plot but it sounds like the tension aspect is pretty cool. Definetly watching it tonight

Re: Gradual Disempowerment: How Even Incremental AI Progress Poses Existential Risks

#73

Post by ADeerAppeared »

cyrillite wrote: post #70
ADeerAppeared wrote: post #21

Earlier quoted context omitted.

> It feels like they always anthropomorphize AI as some sort of "God". It's not like that. It is that. They're playing Pascal's Wager against an imaginary future god. The most maddening part is that the obvious problem with that has been well identified by those circles, dubbed "Pascal's Mugging", but they're still rambling on about "extinction risk" whilst disregarding the very material ongoing issues AI causes. The…

Can you explain how pascal’s mugging functions with respect to risk rather than reward?

In what sense?

The "Mugging" going on is that "AI safety" folks proclaim that AI might have an "extinction risk" or infinite-negative outcome.

And they proclaim that therefore, we should be devoting considerable resources (i.e. on the scale of billions) to avoiding that even if the actual chance of this scenario is minimal to astronomically small. "ChatGPT won't kill us now, but in 1000 years it might" kinda shit. For some this ends with "and therefore you need to approve my research funding application", for others (including Altman) it has mutated into "We must build AGI first because we're the only people who can do it without destroying the world".

The problem is that this is absurd. They're focussing on a niche scenario whilst ignoring horrific problems caused in the here and now.

"Skynet might happen in Y3K" is no excuse to flood the current internet with AI slop, create a sizeable economic bubble, seek to replace entire economic sectors with outsourced "Virtual" employees, and perhaps most ethically concerning of all: create horrific CSAM torment nexuses where even near-destitute gig economy workers in Kenya walk out of the job.

Yet "AI safety" folks would have you believe so.

Re: Gradual Disempowerment: How Even Incremental AI Progress Poses Existential Risks

#74

Post by gom_jabbar »

Jordan-117 wrote: post #63

This just underscores the feeling that most of the problems people have with AI are actually problems with rampant capitalism. Negative externalities, regulatory capture, the tragedy of the commons -- AI merely intensifies them. I've heard it said that corporations are in many ways the forerunners of the singularity, able to act with superhuman effectiveness and marshall resources on a world-altering scale in pursuit…

I think it not only feels bleak, but that it (that capitalism is an it is important to Nick Land from the perspective of complex adaptive systems) is just realistic.

I have a research project on Nick Land's core thesis that capitalism is AI. If you want to go ultra-deep into his theory, check it out: https://retrochronic.com/

It's fundamental to understand that capital is not just teleological - converging with AI on the event horizon of the Singularity - but teleoplexic (i.e. "capitalism takes on a mind of its own and becomes autonomous").

Re: Gradual Disempowerment: How Even Incremental AI Progress Poses Existential Risks

#75

Post by tim333 »

ADeerAppeared wrote: post #73
cyrillite wrote: post #70

Earlier quoted context omitted.

Can you explain how pascal’s mugging functions with respect to risk rather than reward?

In what sense? The "Mugging" going on is that "AI safety" folks proclaim that AI might have an "extinction risk" or infinite-negative outcome. And they proclaim that therefore, we should be devoting considerable resources (i.e. on the scale of billions) to avoiding that even if the actual chance of this scenario is minimal to astronomically small. "ChatGPT won't kill us now, but in 1000 years it might" kinda shit. Fo…

The people who say it's absurd tend to be the least informed while the people saying it's a major risk include the guy who got a Nobel prize for inventing the current stuff and the leading researchers. Here's some names in the field. 15/19 think the risk is significant https://x.com/AISafetyMemes/status/1884562099612889106/photo...

SMBC is quite funny on the AI risks eg. https://www.smbc-comics.com/comic/signal-2

Re: Gradual Disempowerment: How Even Incremental AI Progress Poses Existential Risks

#76

Post by tim333 »

alephnerd wrote: post #9

While this is a well written paper, I'm not sure it's really contextualizing realistic risks that may arise from AI. It feels like a lot of "Existential AI Risk" types are divorced from the physical aspects of maintaining software - eg. your model needs hardware to compute, you need cell towers and fiber optic cables to transmit. It feels like they always anthropomorphize AI as some sort of "God". The "AI Powered Sta…

I think the more realistic threat model is from existing malware and similar stuff. The same people who write viruses, rootkits, try to hack elections, sever communication cables and so on will probably try to do evil AI. Then instead of Putin trying to take over the world and build torture chambers for the resisters you'll have a Putinbot trying to do so for ever more, pre-armed with nukes.

Re: Gradual Disempowerment: How Even Incremental AI Progress Poses Existential Risks

#77

Post by xpe »

alephnerd wrote: post #16
xpe wrote: post #12

Earlier quoted context omitted.

How many people have you talked to face to face about various existential risk scenarios? Have you gotten into probabilities and the logic involved? That’s what I’ve done and this is the level of rigor that is table stakes for calculating the cost-benefit over different possible outcomes.

> How many people have you talked to face to face about various existential risk scenarios A decent amount. I started off as a NatSec adjacent staffer and have helped build Cybersecurity and Defense Tech companies, so a lot of my peers and friends are working directly on US-China, US-Russia, Israel-Iran, Saudi-Iran, Saudi-Turkey, India-Pakistan, and India-China relations. These are all relations that could explode in…

Maybe you are right in predicting (guessing?) as to which is more likely. Still, we don’t have the luxury of just rank ordering failure modes and only mitigating the first.

Not to mention that the race for AI technology is likely going to make geopolitics more volatile. If something happens, it doesn’t matter how we bucketed it conceptually. Reality doesn’t care about where we draw the lines.

The false dichotomies abound in many of these discussions.

Re: Gradual Disempowerment: How Even Incremental AI Progress Poses Existential Risks

#79

Post by marstall »

yapyap wrote: post #4
marstall wrote: post #2

or ... our minds and bodies will quite rapidly adapt!

we as humans evolve, yes. BUT: 1. we evolve quickly in the grand scheme of the universe but slow when talking about human lives. 2. what would rapidly adapting to AI mean? if it’s become reliant on it for the most basic tasks in the way that people nowadays have become so reliant on calculators they barely even bother with doing math in their head, sure. If you mean adapting alongside AI where we in some way would al…

we adapted to the rise of language, the industrial revolution, automation of agriculture, roads and cars, all the previous forms of computer-driven automation ... all of which made some parts of us redundant. and which taken together left us richer, safer and freeer. AI will probably come a few signal benefits (an AI doctor for every person on the planet?) and plenty of sludge and pollution - and we will adapt to that in many ways.

Re: Gradual Disempowerment: How Even Incremental AI Progress Poses Existential Risks

#80

Post by ADeerAppeared »

tim333 wrote: post #75
ADeerAppeared wrote: post #73

Earlier quoted context omitted.

In what sense? The "Mugging" going on is that "AI safety" folks proclaim that AI might have an "extinction risk" or infinite-negative outcome. And they proclaim that therefore, we should be devoting considerable resources (i.e. on the scale of billions) to avoiding that even if the actual chance of this scenario is minimal to astronomically small. "ChatGPT won't kill us now, but in 1000 years it might" kinda shit. Fo…

The people who say it's absurd tend to be the least informed while the people saying it's a major risk include the guy who got a Nobel prize for inventing the current stuff and the leading researchers. Here's some names in the field. 15/19 think the risk is significant https://x.com/AISafetyMemes/status/1884562099612889106/photo... SMBC is quite funny on the AI risks eg. https://www.smbc-comics.com/comic/signal-2

It's called absurd not because it's not understood, or because there aren't technical counterarguments to be made.

It's called absurd because it does not deserve to be humoured the effort of writing out those arguments.

> Here's some names in the field. 15/19 think the risk is significant

A list that is largely a pile of clowns and morons, many with direct financial interests in amplifying the "danger"/power of AI.

This is why the doomsday cult is not taken serious.

Post reply on HN