Earlier quoted context omitted.
There are lots of things AI could do. But the things AI will do are governed by the incentives of the underlying system (i.e. society).
Those incentives are themselves the output of a variety of heuristics that we rely on only due to a lack of viable alternatives. I expect that eventually AI will make some alternatives viable because the bottleneck is so often an information processing one. Three examples: - Loans get issued when they're likely to make a banker rich. We know that's a lousy proxy for assurances that their associated ventures will be b…
This system exists and is optimized to make the bankers rich. The idea that it helps ensure that ventures using that money will be successful is the thin veneer used 100+ years ago to sell it to everyone. But the true purpose has always been to make bankers rich. If you to institute any other system that would not achieve that purpose, you will find yourself battling enormous opposition.
The same things are true for huge parts of our society. Perhaps the most glaringly obvious is the US Healthcare system. Experience from all over the world shows clearly that it is not an efficient way to organize Healthcare, by any stretch of the imagination. Still, it won't change, not because people don't believe the outside examples, but because the system is working for what it was designed to do - transfer huge amounts of money to rich people. And it delivers just enough health care that people aren't routing in the streets against it.