Machine intelligence, part 1

I think this is a more legitimate concern than the fear of the "Matrix outcome" that some people seem to have.

But, what you're describing is the process of people being replaced by technology. Generally speaking, this will probably not be a problem for a free-market economy, although it will certainly result in some unemployment.

The key point is that replacing humans with machines does not only cause unemployment, but it also reduces cost of production, which stimulates capital investment and/or reduces prices in the industry in question. In the general economy, this reduced cost and increased production should offset the lost "purchasing power" from the now-unemployed parties. The stimulus reduces costs in other industries which promotes job growth.

The end result is likely to be that the same number of people are employed, but they are employed in a more efficient manner of production. The cost of labor will decrease relative to the amount of production, however because this is tied to a decrease in cost of production, the real value of the labor (in terms of things you can buy) should not decrease.

Of course there will be some disenfranchised individuals, especially those who have particular skills that are replaced my mechanization. However, this is more likely to affect skilled laborers (like cooks) than those who are not paid for their skills (like janitors or cashiers).

In the end, I guess my point is that a free-market economy naturally balances these factors due to the relationship between supply and demand. However, it's possible that we will reach a point, especially if we truly do hit a Singularity, where we will have to reconsider the use of a scarcity-based economy at all, as production becomes completely divorced from any human action. Hopefully though, at that point the cost of goods will have naturally fallen to such a degree that the transition can be performed peacefully.

  in an effort to accomplish some other goal (most goals, 
  if you think about them long enough, could make use of 
  resources currently being used by humans) wipes us out

This is a line of reasoning put forward a lot, not only in reference to SMIs but also extraterrestrial entities (two concepts that actually have a lot in common), most notably by Stephen Hawking. We're instinctively wired to worry about our resources and like to think their value is universal. It's based on the assumption that even for non-human organisms, the Earth is the end-all-be-all prize. Nobody seems to question this assumption, so I will.

I posit there is nothing, nothing at all on Earth that couldn't be found in more abundance elsewhere in the galaxy. Also, Earth comes with a few properties that are great for humans but bad for everybody else: a deep gravity well, unpredictable weather and geology, corrosive atmosphere and oceans, threats from adaptive biological organisms, limited access to energy and rare elements.

There may well be reasons for an antagonistic or uncaring intelligence to wipe us all out, and an unlimited number of entities can be imagined who might do so just for the heck of it, but a conflict over resources seems unlikely to me. A terrestrial SMI starved for resources has two broad options to consider: sterilize the planet and start stripmining it, only to bump up against the planet's hard resource limitations soon after - or launching a single rocket into space and start working on the solar system, with a clear path to further expansion and a greatly reduced overall risk.

One other thing I'd like to comment on is this idea that an SMI has to be in some way separate from us. While it's absolutely possible for entities to develop that have no connection with humanity whatsoever, I think we're discounting 99% of the rest of the spectrum. It starts with a human, moving on to a human using basic tools, and right now we're humans with advanced information processing. I do not have the feeling that the technology I live my daily life with (and in) is all that separate from me. In a very real sense, I am the product of a complex interaction with my brain as the driving factor, but including just as essentially the IT I use.

When discussing SMI, this question of survival might have a shades-of-grey answer as well. To me, survival of the human mind does not mean "a continued, unmodified existence of close-to-natural humans on Earth". That's a very narrow-minded concept of what survival is. I think we have a greater destiny open to us, completing our long departure from the necessities of biology which we have begun millennia ago. We might fuse with machines, becoming SMIs or an integral component of machine intelligences. I think this is a worthwhile goal, and it's an evolutionary viable answer to the survival problem as well. It's in fact the only satisfying answer I can think of.

I have a feeling this is a leakage of Eliezer Yudkowsky's opinions into tech entrepreneur circles via his friendship with Peter Thiel.

> One of my top 4 favorite explanations for the Fermi paradox is that biological intelligence always eventually creates machine intelligence, which wipes out biological life and then for some reason decides to makes itself undetectable

That's not an explanation of the fermi paradox, that's just moving the problem around. The SMI has no more reason to make itself undetectable than the guys that built it.