Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Gradual Disempowerment: How Even Incremental AI Progress Poses Existential Risks

arxiv.org

Post reply on HN
Print view View original

51–60 of 87 posts

Re: Gradual Disempowerment: How Even Incremental AI Progress Poses Existential Risks

#51

Post by mrob »

tac19 wrote: post #47
mrob wrote: post #44

Earlier quoted context omitted.

But people talking about tools usually mean tools in the concrete sense. Even very complicated tools in the concrete sense, e.g. compilers or CNC mills, only act as multipliers to the operator's ability. A consultant is an agent, not a tool, and this is a better comparison for modern AI. It's an important distinction because an agent can fail or act against your interests through no fault of your own. The ability mul…

Well I don't think getting hung up on such definitions will be fruitful. But here is the point i was trying to make: humans, as individuals and as collectives, do indeed have a lot of experience outsourcing intellectual jobs. They do this knowing full well that the "expert" they're employing is not a deterministic box, and may in fact be secretly working against their interests. None of those problems or potential is…

The human agent has a physical body like yours and shares your evolutionary history. They have reasons to care about things like reputation and social status. An AI agent only "cares" about maximizing or minimizing a number. It's much more difficult to determine if this aligns with your interests.

Re: Gradual Disempowerment: How Even Incremental AI Progress Poses Existential Risks

#52

Post by brandall10 »

phkahler wrote: post #46
alephnerd wrote: post #16

Earlier quoted context omitted.

> How many people have you talked to face to face about various existential risk scenarios A decent amount. I started off as a NatSec adjacent staffer and have helped build Cybersecurity and Defense Tech companies, so a lot of my peers and friends are working directly on US-China, US-Russia, Israel-Iran, Saudi-Iran, Saudi-Turkey, India-Pakistan, and India-China relations. These are all relations that could explode in…

>> I think the world is more likely to end due to bog standard conflicts escalating into an actual war. Not some sort of AGI/ASI going skynet WOPPER, not skynet. You saw War Games right?

It's WOPR = War Operation Plan Response.

Re: Gradual Disempowerment: How Even Incremental AI Progress Poses Existential Risks

#53

Post by bryanrasmussen »

Saline9515 wrote: post #17
alephnerd wrote: post #9

While this is a well written paper, I'm not sure it's really contextualizing realistic risks that may arise from AI. It feels like a lot of "Existential AI Risk" types are divorced from the physical aspects of maintaining software - eg. your model needs hardware to compute, you need cell towers and fiber optic cables to transmit. It feels like they always anthropomorphize AI as some sort of "God". The "AI Powered Sta…

AI powered state would be a significant improvement in many places with endemic corruption. An AI has no incentive to steal.

the corrupt statesman discovered that after the AI which was trained on him was put into place he got the same amount of money delivered into his Swiss bank accounts, as the AI was just taking bribes and depositing them as the training dictated.

Re: Gradual Disempowerment: How Even Incremental AI Progress Poses Existential Risks

#54

Post by whodidntante »

Once PRISM becomes R/W, you will not even know if what you read/hear/see on the internet is actually what others have written/said/created. You will interact with the world as the government wants you to, tailored to each individual.

Each time we choose to allow an AI to "improve" what we write/create, each time we choose to allow AI to "summarize" what we read/consume, we choose to take another step along this road. Eventually, it will be a simple "optimization" to allow AI to do this on a protocol level, making all of our lives "easier" and more "efficient"

Of course, I am not sure if anyone will actually see this comment, or if this entire thread is an AI hallucination, keeping me managed and docile.

Re: Gradual Disempowerment: How Even Incremental AI Progress Poses Existential Risks

#55

Post by alephnerd »

lemonberry wrote: post #45
alephnerd wrote: post #9

While this is a well written paper, I'm not sure it's really contextualizing realistic risks that may arise from AI. It feels like a lot of "Existential AI Risk" types are divorced from the physical aspects of maintaining software - eg. your model needs hardware to compute, you need cell towers and fiber optic cables to transmit. It feels like they always anthropomorphize AI as some sort of "God". The "AI Powered Sta…

Kevin Kelly was on a recent episode of the Farnham Street podcast and was surprised by how much he anthropomorphized AI. Maybe that's not a surprise to people that know more about him, but it was to me.

I think the user experience of seeing a model extruding realistic sounding text is what broke a lot of people.

Back in HS, I introduced a buddy of mine to ELIZA mode in emacs, and it completely broke her mind the same way LLMs did for a lot of people - like she actually conversed with ELIZA and used it to solve her anxiety during the college admissions process. Yet ELIZA used very simple heuristics to extrude human sounding text. And my friend wasn't some dummy or Luddite - she ended up going to an Ivy to study economics and medicine.

It goes to show that User Experience is all that really matters for technology.

Re: Gradual Disempowerment: How Even Incremental AI Progress Poses Existential Risks

#56

Post by ctoth »

alephnerd wrote: post #13
llamaimperative wrote: post #11

Earlier quoted context omitted.

Yeah it would only be worrisome if there were significant incentives to enlist more hardware, cell towers, and fiber optic cables to create and operate increasingly powerful AI and to improve its ability to act directly on the physical world

It's not even about that. That isn't even how a model is created. And that's what I'm getting it. It's basically bad science fiction being used to create some form of a pseudo-religious experience. At least your local Mandirs, mosques, churches, synagogues, gurdawara, and other formal religions do food drives and connect a subset of your local community. What do AI Doomers do? Nothing.

> What do AI Doomers do? Nothing.

You seem very keyed up about this, so it is a good thing you are wrong. AI doomers are far more likely people to be giving to GiveWell, or actually thinking about how they are helping society. Source: Am AI doomer. Donate to GiveWell.

Re: Gradual Disempowerment: How Even Incremental AI Progress Poses Existential Risks

#57

Post by tac19 »

mrob wrote: post #51
tac19 wrote: post #47

Earlier quoted context omitted.

Well I don't think getting hung up on such definitions will be fruitful. But here is the point i was trying to make: humans, as individuals and as collectives, do indeed have a lot of experience outsourcing intellectual jobs. They do this knowing full well that the "expert" they're employing is not a deterministic box, and may in fact be secretly working against their interests. None of those problems or potential is…

The human agent has a physical body like yours and shares your evolutionary history. They have reasons to care about things like reputation and social status. An AI agent only "cares" about maximizing or minimizing a number. It's much more difficult to determine if this aligns with your interests.

Maybe. But a human agent also has personal needs, desires, and self-interests that may motivate treachery. Human's have an evolutionarily proven propensity for duplicity and deceit. It may turn out that some silicon experts are more neutral and less prone to betrayal, and are worthy of trust.

Re: Gradual Disempowerment: How Even Incremental AI Progress Poses Existential Risks

#58

Post by tiborsaas »

If we are speculating on existential risks, then consider Satya Nadella's take on the future of software: https://www.youtube.com/watch?v=a_RjOhCkhvQ

It's quite creepy that in his view all tools, features, access and capabilities will be accessible to an AI agent which can just do the task. This sounds fine in a narrow scope, but if it's deployed at the scale of Windows, then it suddenly becomes a lot more scary. Don't just think of the home users, but businesses and institutions will be running these systems.

The core problem is that we can't be sure what a new generation of AI models will be capable of after a few years of iteration. They might find it trivial to control our machines which can provide them unprecedented access to follow an agenda. Malware exists today to do this, but they can be spotted, isolated and analyzed. When the OS by design is welcoming these attacks there's nothing we can do probably.

But please tell me I've consumed too many sci-fi.

Re: Gradual Disempowerment: How Even Incremental AI Progress Poses Existential Risks

#59

Post by llamaimperative »

alephnerd wrote: post #13
llamaimperative wrote: post #11

Earlier quoted context omitted.

Yeah it would only be worrisome if there were significant incentives to enlist more hardware, cell towers, and fiber optic cables to create and operate increasingly powerful AI and to improve its ability to act directly on the physical world

It's not even about that. That isn't even how a model is created. And that's what I'm getting it. It's basically bad science fiction being used to create some form of a pseudo-religious experience. At least your local Mandirs, mosques, churches, synagogues, gurdawara, and other formal religions do food drives and connect a subset of your local community. What do AI Doomers do? Nothing.

I also wasn’t commenting on how a model is created…?

Do you disagree there are significant incentives to scale up the power, ubiquity, and direct physical impact of these systems?

Please answer with yes or no, and not some snarky “bad science fiction” straw man.

Re: Gradual Disempowerment: How Even Incremental AI Progress Poses Existential Risks

#60

Post by alephnerd »

llamaimperative wrote: post #59
alephnerd wrote: post #13

Earlier quoted context omitted.

It's not even about that. That isn't even how a model is created. And that's what I'm getting it. It's basically bad science fiction being used to create some form of a pseudo-religious experience. At least your local Mandirs, mosques, churches, synagogues, gurdawara, and other formal religions do food drives and connect a subset of your local community. What do AI Doomers do? Nothing.

I also wasn’t commenting on how a model is created…? Do you disagree there are significant incentives to scale up the power, ubiquity, and direct physical impact of these systems? Please answer with yes or no, and not some snarky “bad science fiction” straw man.

While I do think there is an incentive to scale up physical infrastructure, there is a lot of "AI Washing" happening in the space, with bog standard energy projects being justified for "AI Scaling", especially because ESG as a investment category is dead, and a lot of energy investments would be bucketed under the ESG asset class.

> I also wasn’t commenting on how a model is created…?

That comment wasn't targeted at you. I just find AI washing arguments laughable sometimes. It's very similar to the ambulance chasing and scare tactics that is a major part of Cybersecurity GTM.

Post reply on HN