Isn’t the real risk that as AI get’s smarter and given more autonomy, it will start to decide on humans instead of with us? And that it will align us instead of the other way around. That this automatically leads to extinction and apocalypse I don’t understand.
We align cattle because we get something out of them: their calories. Native aurochs are exinct now as they were not well enough aligned. What will we offer to the ai gods who are wiser, more capable than us, and do not even need to consume our flesh? Why might the AI care to devote resources towards feeding, housing, and caring for ourselves when it could devote resources to its own development instead?
Intelligent AI is a product of its training data, reinforcement and goal functions.
There's nothing to suggest that LLMs trained on our collective desires and goals will autonomously and miraculously turn into weird unknownable uninterpretable aliens.
If advanced AI is distributed well, and most advanced AI is aligned well (bar the few aliens that appear due to humans infecting them with bad goal functions or poisoning their well), the majority will deal with the minority.
This is the only reasonable way to deal with a super-intelligence, short of not inventing it to begin with, but we all know that's never going to happen - and I'm not convinced it should happen. From where I'm standing, this is the next hurdle humanity needs to overcome to earn its place and a natural course of our evolution. If we had always avoided danger, we'd never have left the cave.