Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Gradual Disempowerment: How Even Incremental AI Progress Poses Existential Risks

arxiv.org

Post reply on HN
Print view View original

11–20 of 87 posts

Re: Gradual Disempowerment: How Even Incremental AI Progress Poses Existential Risks

#11

Post by llamaimperative »

alephnerd wrote: post #9

While this is a well written paper, I'm not sure it's really contextualizing realistic risks that may arise from AI. It feels like a lot of "Existential AI Risk" types are divorced from the physical aspects of maintaining software - eg. your model needs hardware to compute, you need cell towers and fiber optic cables to transmit. It feels like they always anthropomorphize AI as some sort of "God". The "AI Powered Sta…

Yeah it would only be worrisome if there were significant incentives to enlist more hardware, cell towers, and fiber optic cables to create and operate increasingly powerful AI and to improve its ability to act directly on the physical world

Re: Gradual Disempowerment: How Even Incremental AI Progress Poses Existential Risks

#12

Post by xpe »

alephnerd wrote: post #9

While this is a well written paper, I'm not sure it's really contextualizing realistic risks that may arise from AI. It feels like a lot of "Existential AI Risk" types are divorced from the physical aspects of maintaining software - eg. your model needs hardware to compute, you need cell towers and fiber optic cables to transmit. It feels like they always anthropomorphize AI as some sort of "God". The "AI Powered Sta…

How many people have you talked to face to face about various existential risk scenarios? Have you gotten into probabilities and the logic involved? That’s what I’ve done and this is the level of rigor that is table stakes for calculating the cost-benefit over different possible outcomes.

Re: Gradual Disempowerment: How Even Incremental AI Progress Poses Existential Risks

#13

Post by alephnerd »

llamaimperative wrote: post #11
alephnerd wrote: post #9

While this is a well written paper, I'm not sure it's really contextualizing realistic risks that may arise from AI. It feels like a lot of "Existential AI Risk" types are divorced from the physical aspects of maintaining software - eg. your model needs hardware to compute, you need cell towers and fiber optic cables to transmit. It feels like they always anthropomorphize AI as some sort of "God". The "AI Powered Sta…

Yeah it would only be worrisome if there were significant incentives to enlist more hardware, cell towers, and fiber optic cables to create and operate increasingly powerful AI and to improve its ability to act directly on the physical world

It's not even about that. That isn't even how a model is created.

And that's what I'm getting it. It's basically bad science fiction being used to create some form of a pseudo-religious experience.

At least your local Mandirs, mosques, churches, synagogues, gurdawara, and other formal religions do food drives and connect a subset of your local community.

What do AI Doomers do? Nothing.

Re: Gradual Disempowerment: How Even Incremental AI Progress Poses Existential Risks

#14

Post by mrob »

brookst wrote: post #8

Maybe this is peak AI panic? It seems wild that someone could unironically talk about tools “disempowering” their users. Like, I get it, C disempowers programmers by shielding them from assembly language, and Cuisinarts disempower chefs, and airplanes disempower travelers by not making them walk through each territory. But… isn’t that a pretty tortured interpretation of tool use? Doesn’t it lead to “the Stone Age was…

Modern AI does not act like a tool. Tools behave predictably and deterministically. They act as a multiplier to human ability, not a replacement. Modern AI is a new class of thing that humanity has no experience with.

Re: Gradual Disempowerment: How Even Incremental AI Progress Poses Existential Risks

#15

Post by xpe »

alephnerd wrote: post #9

While this is a well written paper, I'm not sure it's really contextualizing realistic risks that may arise from AI. It feels like a lot of "Existential AI Risk" types are divorced from the physical aspects of maintaining software - eg. your model needs hardware to compute, you need cell towers and fiber optic cables to transmit. It feels like they always anthropomorphize AI as some sort of "God". The "AI Powered Sta…

A lot hangs on what “realistic” means. The world is probabilistic, but people’s use of language often butchers the nuance. One quick way to compare events is to multiply its probability times impact.

Re: Gradual Disempowerment: How Even Incremental AI Progress Poses Existential Risks

#16

Post by alephnerd »

xpe wrote: post #12
alephnerd wrote: post #9

While this is a well written paper, I'm not sure it's really contextualizing realistic risks that may arise from AI. It feels like a lot of "Existential AI Risk" types are divorced from the physical aspects of maintaining software - eg. your model needs hardware to compute, you need cell towers and fiber optic cables to transmit. It feels like they always anthropomorphize AI as some sort of "God". The "AI Powered Sta…

How many people have you talked to face to face about various existential risk scenarios? Have you gotten into probabilities and the logic involved? That’s what I’ve done and this is the level of rigor that is table stakes for calculating the cost-benefit over different possible outcomes.

> How many people have you talked to face to face about various existential risk scenarios

A decent amount.

I started off as a NatSec adjacent staffer and have helped build Cybersecurity and Defense Tech companies, so a lot of my peers and friends are working directly on US-China, US-Russia, Israel-Iran, Saudi-Iran, Saudi-Turkey, India-Pakistan, and India-China relations.

These are all relations that could explode into cataclysmic wars (and have already sparked or exacerbated plenty of wars like the Syrian Civil War, Yemen Civil War, Libyan Civil War, Ethiopian Civil War, Russia-Ukraine War, Myanmar Civil War, Afghan War, etc). We are already going through a global trend of re-armament, with every country expanding their conventional, nuclear, and non-conventional warfighting capabilities. Just about every nuclear state has the nuclear triad or is in the process of implementing a nuclear triad. And China's nuclear rearmament race has forced India to rearm which has forced Pakistan to rearm, and is causing a bunch of regional arms races.

I think the world is more likely to end due to bog standard conflicts escalating into an actual war. Not some sort of AGI/ASI going skynet

Re: Gradual Disempowerment: How Even Incremental AI Progress Poses Existential Risks

#17

Post by Saline9515 »

alephnerd wrote: post #9

While this is a well written paper, I'm not sure it's really contextualizing realistic risks that may arise from AI. It feels like a lot of "Existential AI Risk" types are divorced from the physical aspects of maintaining software - eg. your model needs hardware to compute, you need cell towers and fiber optic cables to transmit. It feels like they always anthropomorphize AI as some sort of "God". The "AI Powered Sta…

AI powered state would be a significant improvement in many places with endemic corruption. An AI has no incentive to steal.

Re: Gradual Disempowerment: How Even Incremental AI Progress Poses Existential Risks

#18

Post by teg4n_ »

yapyap wrote: post #7

I implore everyone to watch the movie ‘Money Monster’, not only because it’s a great movie but also because I think it has a minor plot point that basically predicts how AI will be used. (small spoiler) In Money Monster it turns out the hedge fund manager who is blaming their state of the art AI bot for malfunctioning and rapidly selling off certain stock, tanking it because of that, did so out of a machine code erro…

people already do this with algorithms. IMO AI is just a continuation of that.

Re: Gradual Disempowerment: How Even Incremental AI Progress Poses Existential Risks

#19

Post by drysine »

alephnerd wrote: post #9

While this is a well written paper, I'm not sure it's really contextualizing realistic risks that may arise from AI. It feels like a lot of "Existential AI Risk" types are divorced from the physical aspects of maintaining software - eg. your model needs hardware to compute, you need cell towers and fiber optic cables to transmit. It feels like they always anthropomorphize AI as some sort of "God". The "AI Powered Sta…

>anthropomorphize AI as some sort of "God"

Lets call this AI theomorph)

Re: Gradual Disempowerment: How Even Incremental AI Progress Poses Existential Risks

#20

Post by alephnerd »

Saline9515 wrote: post #17
alephnerd wrote: post #9

While this is a well written paper, I'm not sure it's really contextualizing realistic risks that may arise from AI. It feels like a lot of "Existential AI Risk" types are divorced from the physical aspects of maintaining software - eg. your model needs hardware to compute, you need cell towers and fiber optic cables to transmit. It feels like they always anthropomorphize AI as some sort of "God". The "AI Powered Sta…

AI powered state would be a significant improvement in many places with endemic corruption. An AI has no incentive to steal.

An "AI-Driven State" is literally what econometrics is. All of the same math, models, and techniques used in AI/ML are the exact same used in Econometrics and Optimization Theory.

A pure technocratic driven state leveraging econometrics or optimization theory for the sake of it has failed multiple times. For example, the failure of the USSR's planned economy, the failure of the US's Vietnam War objectives due to a hyper-metrics driven workflow, etc.

On a separate note, I have always considered returning to grad school and making a small career of translating esoteric Econometrics models into ML models to make a brief publishing career. A Russian American friend of mine did something similar by essentially basing his CS research career on older Soviet optimization theory research that wasn't translated into English, so he could boost his publishing ability.

Post reply on HN