I sometimes wonder what freedom really is. Should the freedom to harm others also be guaranteed as freedom? Individual computational sovereignty versus the prevention of social harm is always contentious, and claiming that something is always right is always extremist.
If it is "intelligence," is it not natural to reject that? I also think local models should adapt to me when it comes to safety issues, but people bring up examples that are too extreme.
Programming is the same, and in fact, most problems are boundary problems. It is the things that straddle the boundaries that always make us think. The principles at those moments change every time, shifting with the situation and context. Is that not just a childish way of thinking? Even in programming, just the issue of granting root permissions is enough to cause endless fights.
I agree with the idea early in the text that intelligence is not everything. Intelligence includes bodily intelligence as well, and we lump it all together into one thing, but there is so much of it. The variance in intelligence is vast, and those people also need to be able to live their lives. That is why I think intelligence alone will not solve everything. I too believe that the human species may disappear and an inorganic species could emerge later, but I find it hard to understand why people talk about such extreme risks. And it is not true that making a chip in a semiconductor fab involves almost no human intervention. If you have experience supplying equipment to such fabs, you would know there are quite a few points where humans are involved. Though sure, they could be replaced.
In my view, society is simply worshipping the abstract concept of "intelligence" and projecting its desires onto it. The AGI narrative is just a kind of cargo cult, a projection of capital by the tech elite. Software eating the world, superintelligence solving everything. The masses engage in messianic projection, and tech companies, facing declining growth engines in their own businesses, are trying to create new ventures to pour it all into. A market that is large enough becomes too big to sustain massive growth rates every time, and when growth rates are that high, the larger the company, the more its sector's growth rate tends to converge with its own. This is usually called the law of large numbers. The problem is that CEOs and these entrepreneurs always want growth rates above a certain threshold, so they are simply searching for new pastures. AGI is just being pumped up out of financial necessity.
Capital will create gravity and bring forth new technologies. That is the allure of capital, after all.
But that does not mean all problems will be solved, and inequality will deepen. Only the distribution of power will shift.