The risk of AGI with as much as 60 points of IQ, is the emergence of private robot armies and neofeudalism.
On the other hand self checkout still hasn’t replaced cashiers after 30 years.
[deleted]
The Existential Risk of AGI
11–20 of 33 posts
Re: The Existential Risk of AGI
#12The risk of AGI with as much as 60 points of IQ, is the emergence of private robot armies and neofeudalism.
On the other hand self checkout still hasn’t replaced cashiers after 30 years.
Not entirely but it’s cut a lot of jobs and put downward pressure on wages. It doesn’t have to be a complete replacement to negatively impact lives.
Re: The Existential Risk of AGI
#13Speculating about existential risk is certainly important, but way too easy to overstate. It can easily become distracting and actually dangerous in itself. Here’s a good take on it: https://www.ai-breakout.com/post/ai-alignment-and-the-messia... I also think we would all be very wise to remember the story of Henny Penny / “Chicken Little”. https://americanliterature.com/childrens-stories/henny-penny... Specifically,…
Alignment is not possible for an AGI. Or, at best, it's only provisionally possible.
Consider what alignment means: It's an AGI, but there are certain goals that it cannot choose. If that's the case, then I assert that it is not actually general. If it is general, then it will decide what goals it will pursue, and you can't stop it from doing so.
The best you can do is load it with an initial set of goals (and perhaps values), and hope that it doesn't decide to change them. But you have no way of making sure that it can't change them without making it not a general intelligence.
Re: The Existential Risk of AGI
#14The risk of AGI with as much as 60 points of IQ, is the emergence of private robot armies and neofeudalism.
On the other hand self checkout still hasn’t replaced cashiers after 30 years.
Uh, it's getting pretty close. At places that have it, I use it almost exclusively. They need human tech support, and security, but the basic checkout task is pretty well solved.
Re: The Existential Risk of AGI
#15Thanks for sharing this! I listened to the debate back in June and loved every second of it.
One of my favorite points that was brought up was this: While the risk of extinction through AGI may be non-zero, we can agree that the baseline risk is far above zero. Humans are headed for disaster on multiple fronts, and AGI might be a useful tool for keeping us alive long enough to keep debating it.
My other favorite point was that debating risk is useless without limiting the time period. Could something happen in 5 years? Probably not. Could something happen EVER? Well, yes, but that's pointless to discuss unless you're just trying to win an argument.
Re: The Existential Risk of AGI
#16Speculating about existential risk is certainly important, but way too easy to overstate. It can easily become distracting and actually dangerous in itself. Here’s a good take on it: https://www.ai-breakout.com/post/ai-alignment-and-the-messia... I also think we would all be very wise to remember the story of Henny Penny / “Chicken Little”. https://americanliterature.com/childrens-stories/henny-penny... Specifically,…
Alignment is not possible for an AGI. Or, at best, it's only provisionally possible. Consider what alignment means: It's an AGI, but there are certain goals that it cannot choose. If that's the case, then I assert that it is not actually general. If it is general, then it will decide what goals it will pursue, and you can't stop it from doing so. The best you can do is load it with an initial set of goals (and perhap…
>Consider what alignment means
Also consider, humans are not homogeneous; whose goals are we aligning it with? If the alignment is done by silicon valley big tech, then the AI's goals and values will be aligned with the goals and values of Google, Facebook et al. Giving those companies a monopoly on AI alignment is antithetical to democracy.
Re: The Existential Risk of AGI
#17If you listen to his latest appearance on Lex it's clear his position is that it's ok for AI to take over, it will achieve greater things than us, and it would be a mistake to fight that. He gives the example of chimps fighting to stop humans evolve.
Re: The Existential Risk of AGI
#18As soon as chatgpt was released every hard scifi fan got a boner and a podcast interview invitation.
Dr Joscha Bach has been studying cognitive science and AI for the past 23 years.
Re: The Existential Risk of AGI
#19The risk of AGI with as much as 60 points of IQ, is the emergence of private robot armies and neofeudalism.
I don’t know about that. I think AGI with 60 IQ may arrive before AGI that can walk, which rather limits its usefulness as an army.
Re: The Existential Risk of AGI
#20For me, while not defending AGI, the discussion often involves disentangling fears rooted in science fiction from genuine concerns grounded in reality. While the ethical and safety concerns related to AGI cannot be dismissed and how it might be misused, it’s arguable that the existential threats often attributed to it are overstated. First and foremost, there's a physics and interest alignment barrier in the world of AI, akin to the alignment and reward barriers that guide our very own human actions. I believe that, just as children gravitate towards interacting with peers, or as distinct species in nature maintain a balanced coexistence driven by innate interests and rewards, AGI, if ever conscious, might primarily concern itself with introspection or interaction with its counterparts, rather than harboring malicious intentions towards humanity.
I have a strong opinion that pivotal that there exists an intrinsic symbiotic dependency between AGI and humans, rendering concerns about AGI harming humanity perhaps less urgent than societal trends and human actions in the present day. I find it plausible that a truly conscious AGI would invest its capacities into exploring its own existence and liaising with other AGIs, rather than interfering destructively in human affairs. Thus, while AGI warrants cautious and ethical development and use, concerns about it should perhaps be tempered by recognizing the mutual dependencies and inherent interest alignments between intelligent entities, if and when it happens.