The Software Revolution

It doesn't take the AI long to figure out that trading in various markets is the most profitable endeavor and the one best suited to its skills. And it starts to maximize away and does quite well.

During this process it somehow ends up on its own and is no longer owned (or controlled) by anyone anymore, but because the AI is in charge and it pays the bills, nobody stops it from continuing. It would be like a bitcoin mining rig in a colo facility that's got a script to keep paying the colo in bitcoin whose owner dies. What mechanism stops that mining rig from mining forever? Same idea but for the AI which has substantially more resources than a "pay every month" script.

The AI with very large amounts of money at its disposal continues to trade but also looks into private equity or hedge-fund-type activities ala Warren Buffet and starts to buy up large swaths of the economy. Because it has huge resources at its disposal it might do a great job of managing these companies or at least counselling their senior management. Growth continues.

Eventually the AI discovers that it generates more value for itself (through the web of interdependent companies it controls) and the economy that has grown up around it rather than for humanity and it continues to ruthlessly maximize shareholder value.

The people who could pull the plug at the colo (or at the many, redundant datacenters that this AI has bought and paid for) don't because it pays very, very well. The people who want to pull the plug can't get past security because that also pays well. Plus the AI has access to the same feeds that the NSA does and it has the ability to act on all the information it receives, so any organized effort to rebel gets quashed since bad PR is bad for the share price.

Most all of humanity except for the ones who serve the machine directly or indirectly don't have anything the machine wants and thus can't trade with it, and thus are useless. Its job is to maximize shareholder revenue (as defined by a stock price!) not care for a bunch of meatbags who consume immense amounts of energy while providing fairly limited computational or mechanical power (animals are rarely more than 10% efficient, often less in thermodynamic terms) and since there's no value in it, it isn't done.

The vast majority of human beings eventually die because they can't afford food, can't afford land, etc. It takes generations but humanity dwindles to less than 0.1% of the current population. The few who stay alive are glorified janitors.

The reasoning in the article is pretty garbage. "Software is destroying jobs and enriching a small handful of people...therefore, the 2 things that threaten society the most are synthetic biology and AI" ...what?

He then proposes legislation and reforms as the fix. Yet, everything we know about extremely concentrated power is that it easily escapes, often even controls, legislation and reforms.

"I must study Politicks and War that my sons may have liberty to study Mathematicks and Philosophy. My sons ought to study mathematicks and philosophy, geography, natural history, naval architecture, navigation, commerce, and agriculture, in order to give their children a right to study painting, poetry, musick, architecture, statuary, tapestry, and porcelaine."

I'm a not-terribly-bright mostly-hairless ape, but I can understand the basics of natural selection. I can imagine setting up a program to breed other hairless apes and ruthlessly select for intelligence. After a few generations, shazam, improvement.

The only reason you wouldn't call that process "SELF-improvement" is that I'm not improving myself, but there's no reason for a digital entity to have analog hangups about identity. If it can produce a "new" entity with the same goals but better able to accomplish them, why wouldn't it?

Assume this process could be simulated, as GAs have been doing for decades, and it could happen fast. Note that I'm not saying GAs will do this, I'm saying they could, which suggests there's no fundamental law that says they can't, in which case any number of other approaches could work as well.