The impossibility of intelligence explosion

> Of course you can increase the throughput of a factory by speeding up "the conveyor belt"--a standin for the complex processes going into manufacturing.

You are treating it as a "standin" for something else, when it's unequivocally not. It's one part of a larger process, which is the point he is obviously making. It doesn't make sense when deliberately misinterpreted. Most importantly, intelligence isn't a ladder that can be "sped up" in every dimension.

I don't want that world either. But achieving it is a wet dream for a business.

The challenge that Elon Musk et al are warning us about is what role will humans have after we achieve this dystopia.

The argument hinges on the meaning of intelligence. Indeed, if you consider intelligence to be exactly the thing measured by IQ, an explosion is probably impossible.

Suppose we considered 'effectiveness' -- the ability of a system to quickly achieve its goals. Would the author argue that a recursively self-improving machine could not exponentially increase its effectiveness? Why?

Wouldn't an effectiveness explosion have similar consequences for people -- making human ability nearly irrelevant since any goal we can articulate can be achieved so much faster by machines?

> I don't think the author has any insight at all on what a general AI will be.

To be fair, nobody does. But you're right, the author shouldn't be making these statements with such certainty.

This article is typical of many that claim proven limits on the feasibility (or, in this case, the capabilities) of generalized artificial intelligence, in that it structures the argument in a way designed to avoid discussion of the issue.

It starts by claiming that there is no such thing as general intelligence. What specialized intelligence, then, is human intelligence? It's specialized for "being human". The author is apparently unaware that this tautological response eliminates the distinction between general and specialized intelligence, as one could just as validly (or vacuously) say that a superhuman intelligence is specialized in being what it is and doing what it does. The author has invalidated the hook on which he had hung his argument.

A lot of column-inches are expended on repeatedly restating that animal intelligences have co-evolved with their sensorimotor systems, which is a contingent fact of history, not a fundamental necessity for intelligence (as far as we know; but then the whole article is predicated on the feasibility of AI.) He raises the 'brain in a vat' trope, but no-one is suggesting that AIs must be disconnected from the external world. Furthermore, this line of argument ignores the fact that many of the greatest achievements of human intelligence have come from contemplating abstract ideas.

When the author writes "most of our intelligence is not in our brain, it is externalized as our civilization", he is confusing the achievements of intelligent agents for intelligence itself. When he writes that "an individual brain cannot implement recursive intelligence augmentation" he is confusing a limit on human capabilities for a fundamental limit on intelligence itself...

I am far from convinced that the singularity must follow from achieving human-level artificial intelligence, as we don't know how to get to the starting line, let alone know how the problem of bootstrapping intelligence scales, but the arguments presented here do nothing to persuade me that it is impossible.