71–80 of 374 posts
I've always wondered why people think that an AI that's super-intelligent will also be evil. It could just as likely end up being very kind (more than likely, actually, because the programmers would have safeguard to ensure that it's nice).
there's very few ways to be good and many ways to be evil
AGI escaping a sandbox is truly terrifying. There will be a subgroup of the population that will worship it and work for it. It's not so much AGI that scares me - it's the humans I'm scared of.
AGI escaping a sandbox is truly terrifying. There will be a subgroup of the population that will worship it and work for it. It's not so much AGI that scares me - it's the humans I'm scared of.
Even if a terrifying AGI rises up and escapes our control, I am still eternally grateful that Prometheus stole fire from the gods and gave it to us in the form of technology, knowledge, really civilization.
Life would have continued just fine without many technologies, and I think AGI is one of them. We do not need AGI to survive and grow, and therefore any risk it imposes should be treated from that lens. If the choice is civil collapse, or no AGI, we should pick no AGI. I don't think we would though.
If I let my imagination run wild, I can imagine that at some point AI becomes in some way sentient. By that I mean it gains reasoning, some sort of understanding, and motivation. I wonder what the possibility is that this AI will decide that a pitched battle is going to waste resources and risk humans pulling the plug. What if it understood that and instead operated so subtly that it was not obvious it was controllin…
AGI escaping a sandbox is truly terrifying. There will be a subgroup of the population that will worship it and work for it. It's not so much AGI that scares me - it's the humans I'm scared of.
Ted Chiang has a lovely post about similar ideas, hitting a little closer to home. His thesis AIUI is that unrestrained capitalism, here meaning the unfettered desire to maximize profit for shareholders, is an artificial value system that causes its agents (corporations) to exhibit intelligent goal-seeking behavior. These systems are certainly “alive” — corporations respond to stimuli, sense their environment, “reproduce,” enact changes in response to predictions of future state, etc. Though each of these agents (corporations) are made of collective human behavior, their actions taken together can be considered a form of artificial intelligence that stretches “beyond” (in a wisdom-of-crowds sense) human understanding. In this sense, an artificial intelligence has already “escaped” and has fervent followers.
I know these are strained analogies, but it’s fun to think about. I feel that in the future, the work of solving “AGI safety” will become indistinguishable from the work of other societal problems — how do we prevent tyrants from taking over governments, and how do we make existing governments more resistant to that failure mode? How do we ensure that generating value isn’t a prerequisite for human survival? How can we more efficiently distribute our resources and reduce wealth inequality? How can we ensure that all kinds of life can thrive, not just the most optimal kinds?
If AGI reflects humanity’s best and worst impulses, and I believe it does because that’s all we train it to do, then having good societal answers to these distinctly “human” questions will also help our society resist malevolent AGI. It’s only human, after all.
AGI escaping a sandbox is truly terrifying. There will be a subgroup of the population that will worship it and work for it. It's not so much AGI that scares me - it's the humans I'm scared of.
Isn’t ChatGPT et all already a general intelligence that’s escaped the sandbox ?
AGI escaping a sandbox is truly terrifying. There will be a subgroup of the population that will worship it and work for it. It's not so much AGI that scares me - it's the humans I'm scared of.
Isn’t ChatGPT et all already a general intelligence that’s escaped the sandbox ?
AGI escaping a sandbox is truly terrifying. There will be a subgroup of the population that will worship it and work for it. It's not so much AGI that scares me - it's the humans I'm scared of.
I think this point is underrated, but I’d press further. How do we know this has not already happened? Agencies do not need to be born of gradient descent and certainly do not need to look like matrix multiplications to be deemed “artificially intelligent.” Religion is one such system for example, as are cults, addictions, brands, war strategies, organizing campaigns, governments, or (borrowing from chaos magick) any…
In any doom scenario, like let's say >90% of humans dead, who runs the power plants that supply the AI data centers? Who runs the fabs producing more chips? Who maintains the plumbing? Presumably all these systems fall over and die within a few weeks of the AI deciding to wipe us out. Then what? The AI then dies too. It seems absurd to me that any planner capable of effortlessly destroying humanity would not see that…
A very capable AI can create robots to do those thing better than people can or alternatively create better data centers and fabs that run themselves without human help.
Or do you just not believe that if AI research continues unimpeded the way it has been that eventually researchers will create an AI that is better at inventing technologies than humanity is? I mean, it might take a few decades, but that is where this is heading as far as I can tell.
In any doom scenario, like let's say >90% of humans dead, who runs the power plants that supply the AI data centers? Who runs the fabs producing more chips? Who maintains the plumbing? Presumably all these systems fall over and die within a few weeks of the AI deciding to wipe us out. Then what? The AI then dies too. It seems absurd to me that any planner capable of effortlessly destroying humanity would not see that…
Do you think that your final sentence buys humanity a thousand years, a hundred years, or what?
In the meantime we can figure out either alignment or coexistence.