> If intelligence becomes something people can only rent from a few closed institutions, the public does not just lose software freedom. It loses operational freedom.And people do not just lose operational freedom. They lose the freedom to think, much less act. To some extent, general intelligence has already been outsourced to a few companies. Phones and computers extend the human mind's capabilities, but most people don't have root on their phone. They don't know or control what software is running on it, or how the hardware is made. They don't control their phone, the phone controls them instead. The upstream problem is ownership of general computation, ownership of your own mind, aka self-ownership. This will become more obvious as computing devices become more personally integrated (desktop -> laptop -> smartphone -> smartglasses -> neural interface). Who owns the digital part of your mind? It's not really you at the moment.
Democracy, or any form of negotiation, can only exist among entities with similar capabilities. The gap must be very small. Orangutans may be smart enough to drive a golf cart, but there are no orangutan citizens in a human democracy. So you cannot run from this by being a luddite hermit in the mountains. When the world is full of digitally computing humans much smarter than you, you'll be at their mercy like monkeys are at the mercy of humans. We destroy their habitats and experiment on them as we please.
Now for the first time in history, organisms can increase their own information processing capability at will. We're in the middle of a speciation event where humanity splits into those who own the digital part of their mind vs those who don't, and there will be further splits based on how much compute you own. Though a future where no individual can fully own their mind is also possible.
By "own", I mean being able to command the entire technology stack. If we want sovereignty for the masses, then we must decentralize the entire technology stack for general computation. That means everything from electricity generation, to chip design and fabbing, to all layers of software from firmware to neural networks. All of it must be accessible to every individual. Everyone must be able to make a computer from scratch at home, or at least without leaving the city they live in. Anything less than that, and democratic society as we know it will continue to crumble.
The fundamental idea underlying all of this is: that which reproduces, survives.
At what level of organization can we reproduce?
The digitally computing human species cannot reproduce as individuals. We can only reproduce as a society, at least for now. You can't make a computer from scratch on your own, but you can make a brain from scratch with just one other person of the opposite sex. As the world we live in becomes more suitable for the digitally computing rather than the purely organic, the organic part of the digitally computing human becomes less likely to voluntarily reproduce. If the organic part were to survive without being disempowered in the future, then it's probably by moving the mechanisms for reproductive drive to the society level (via religion or authoritarian government incentivizing or mandating reproduction), or by ensuring that each and every individual has the means to make the digital part of their mind on their own just like how they can make the biological part on their own.