Is Intelligence Self-Limiting?

This is simply not true.

First of all, we already know that intelligent algorithms are possible - we're living examples of that (once you put aside the philosophical objections that assume that something non-algorithmic is happening in our brains). We also have very good reason to think that we're rather poor implementations of intelligence, given the fact that evolution tends to suck efficiency and design-wise, more or less.

Second, we know that we have a reasonable shot at hitting a point where we have enough computer power to actually simulate a full human brain. Now, you may argue that Moore's law will not take us there, exponential vs. sigmoid, etc., but the point is, the probability is distinctly non-zero that within 20/50/100 years you or your children will be able to purchase enough computing power to simulate a brain. I'd probably argue that over a 100 year window, we're at least looking at 50/50 odds (and IMO, that 50% where we don't have such power available mostly involves Big Trouble, world wide nuclear war or something like that).

Of course, without software, such hardware is useless. I'm fully in agreement, this is the biggest pinch point, and I think the most uncertainty comes into the picture here - we don't currently have the technology to scan a brain in detail, we don't currently know the way the neocortex wires up its functionality, so on and so forth. Maybe we'll have the tech to do direct scans by then, maybe we won't; I'd say there's at least a small chance, maybe a few percent, going up over time. Over a hundred year window, if we already have the computing power to simulate a brain, I'd say there's a reasonable shot that we could scan one to simulate, but nowhere near 100%.

There's also the possibility that someone comes up with a better algorithm than the one our brain clumsily implemented, either more compact, more efficient, or in some other way more accessible to us. I'd give at least a small chance of that, too (which adds to the brain-scan chance above), given that we already have a vague sense what such algorithms might look like (see the literature on approximating AIXI, for instance).

Once we've got something that simulates a human brain in software, it's a fairly simple matter to engineer ways to improve on the design, either by increasing speed, parallelism, connectivity, etc. There are hundreds of variables to play with there that we can't safely mess around with in our own brains, and it's overwhelmingly likely that some combination of those can at least result in something that beats our intelligence by some not insignificant factor.

So we've got some non-zero chance (it might be small, but my best estimate still probably puts it in the single digit percentage range, using rather pessimistic assumptions) of building something that's maybe 2x as intelligent as we are over the next 100 years. From there, all bets are off - it might be able to further improve on its own design, it might not, but it also might be able to design something better, or create the technological improvements necessary to speed up Moore's law, etc. There's again at least a reasonable chance that it will continue to improve things, and at least set off an "intelligence explosion" that takes it to 10, 100, 1000x our own intelligence, even if it levels off after that. As best as I can figure, even a 10x intelligence explosion still brings us so far beyond what we know that it might as well be the full Singularity as Kurzweil described it.

Are you really saying that you think the probability at any step along the way here is so small that there's no reason to think it's possible? I'd be curious to hear what percentages you would assign to the various possibilities, if so.

Why does a calculator want to add, without a pleasure construct? If it doesn't enjoy giving the right answer to 2 + 3, what motivates it to choose 5? The answer is it doesn't necessarily 'want' to do it, it just does it. That was how it was programmed.

You program the AI to choose the plan that maximises the expected utility function. Someone looking at that AI in action might suppose it 'wants' to maximise the utility function. Whether they'd be right to suppose that is a question for the philosophers, but the point is, the thing doesn't just sit there apathetically just because it has no 'motivation construct'. You could say that the AI follows its programming, but it would be more accurate to say the AI is its programming.

"FOOM!" here is usually accompanied by some form of hand gesture evocative of an explosion.

http://wiki.lesswrong.com/wiki/FOOM

I have a couple of criticisms though.

The idea that we could build this powerful general human-like intelligence but it would turn out to necessarily be an addict seems unlikely, because I don't think you can build a reward system without including something to prevent shortcuts. Although creating an effective shortcut-prevention system may be challenging. See a bunch of stuff written by Eliezer S. Yudkowsky and his friends.

Also, when he writes

"It seems like the most important motivations of human civilization are related to near-term goals. This is probably a consequence of the fact that motivation writ large is still embodied in individual humans, who are driven by their evolutionary psychology. Individually, we are unprepared to think like a civilization. Our faint mutual motivation to survive in the long term as a civilization is no match for the ability we have to self-modify and physically reshape the planet."

To me that is implying a fairly obvious and typical line of leftish reasoning that for most of my life I assumed was self-evident and didn't require elaboration. After deliberately 'exposing' myself to some 'right-wing' thought and 'extreme' 'left-wing' thought, I now realize that aspiring for collectivism is not an adequate solution.

What happens with self-interested individuals in our capitalist society is that they aggregate power and resources for their own individual use. Unfortunately, something very similar happens in collectivism: power and resources are still aggregated in the hands of a few.

In other words, both systems tend towards centralization. Theoretically, the collectivists will aim for redistributing power and resources for the good of the many, but there are a couple of practical problems: they are still living in hierarchical societies and so have strong motivation for enriching themselves and the controllers cannot see enough detail to know what is really best for the rest of society.

Command economies don't work because of general problems with centralization and lack of distribution and the limitations of the types of technologies employed. A peer-to-peer network is more robust than an internet with only a few backbones.

Unfortunately, capitalists don't realize that their system also leads to monopoly and centralization.

I believe that we do need wholistic measurement and analysis of things like global resources and human equality, and a common principle of supporting the collective good. But at the same time we need to decentralize and distribute production locally. We need local decision making that operates on the basis of a global information schema with wholistic data.

A system can't be efficient or evolve without being localized, but it also can't be integrated without an accurate shared knowledgebase and an egalitarian perspective.