Superintelligence: The Idea That Eats Smart People

For an artificial mind to have anything resembling a human value system, the argument goes, we have to bake those beliefs into the design.

Because we are so nice to the lesser intelligent creatures of our world? We don't even understand our own consciousness, surely our suffering is not very real and we can be used for meat.

So it's super smart and has its own goals. We can reliably presume that it will need energy to achieve those goals. Will it need to achieve them quickly? Why? Would the superintelligence be shortsighted enough to provoke humans into active combat against it? I see no reason to just assume we know the thing would have human eradication as a goal, need exorbitant amounts of energy and resources because it sees achieving its goals as a terribly time-critical thing to do, etc.

Also, if we're going to build a brain and assume no quantum weirdness - why assume the total absence of a subjective morality? Why assume completely immunity to social influence, which the brains we observe most all encounter?

And let's not forget - human beings are in a weird spot, intelligence-wise. We're smart enough to achieve things, but we're not smart enough to be crippled by the profound lack of control we have over things. We're totally comfortable going out and driving around all day, even though we claim to value human life extremely highly and even though we know that causing deaths of innocent human beings is not THAT far-fetched of an outcome of driving a car. I would be unsurprised if we flipped on a super-AI and after 5 minutes it simply stopped in its tracks, having determined that the probability that some action it would take would result in its own destruction with a non-zero probability and that it is instead taking the safe route and not acting at all. No matter how superintelligent it is, it is not going to be able to magically compensate for the influences of chaos theory which destroy the ability to be certain about ANY prediction. We, as humans, feel very clever with our assumed spherical cows, frictionless surfaces, homogenous air pressure and zero wind resistance. Why would a superintelligence also be comfortable with that?

But in practice much of the wealth of super-wealthy people is actually either tide up in the value of the businesses that they own, which are often doing economically valuable things, or is invested in useful enterprises (shares), or funds useful activities (bonds). It's not as though the net wealth of Warren Buffet is all being thrown on hookers and blow.

There are already ways to direct the spending of the wealthy towards more productive uses, such as consumption taxes on luxury goods. But if they take their wealth to other countries with laxer consumption taxes, there's no a lot we can do about it. So we're back to the libertarian argument. At some point you get into questions of freedom and individual rights.