Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The impossibility of intelligence explosion

medium.com

Post reply on HN
Print view View original

21–30 of 143 posts

Re: The impossibility of intelligence explosion

#21

Post by titzer »

> A high-potential human 10,000 years ago would have been raised in a low-complexity environment, likely speaking a single language with fewer than 5,000 words, would never have been taught to read or write, would have been exposed to a limited amount of knowledge and to few cognitive challenges. The situation is a bit better for most contemporary humans, but there is no indication that our environmental opportunities currently outpace our cognitive potential.

More crap. Our environmental opportunities don't outpace our cognitive potential? What internet is the author connected to? The author even goes on in detail in the very next paragraph about the possibilities around...E.g. there are literally thousands, perhaps millions of hours of instructional videos on YouTube, just for basic skills. Every field of human endeavor has uploaded the digital artifacts of its best minds in one form or another to a single global network of unimaginable, fractal, complexity. You don't think you can saturate your cognitive potential in your lifetime given this resource?

Re: The impossibility of intelligence explosion

#22

Post by tome »

smallnamespace wrote: post #15
kamilner wrote: post #4

Earlier quoted context omitted.

Why isn't it possible (or likely, even) that the difficulty of constructing capacity X+1 grows faster than the +1 capacity? Self-improvement would slow exponentially when it takes three times the resources/computation/whatever to construct something that's twice as good at self-improving, for example.

You're arguing that it's not an exponential that doesn't continue to the right indefinitely. But what if it follows a sigmoid instead, but the plateau is much higher than the current level? This is what punctuated equilibria look like -- even if the 'new thing' isn't actually a singularity, it may be enormously disruptive and completely displace whatever came before.

> This is what punctuated equilibria look like -- even if the 'new thing' isn't actually a singularity, it may be enormously disruptive and completely displace whatever came before.

Right, and this is a much more plausible claim than one of "singularity".

Re: The impossibility of intelligence explosion

#23

Post by wdiamond »

This dude don't really understand speed of light multiplied by number of chip factories. Its about self improvement at another speed, without human bureaucracy(fetch). Everybody knows to not run against a car by foot. Its the same. Not really a Doom, but maybe. And for sure not the last tool. Economy will suffer. Kind like having a genius baby that you have "no duties", if it cries too much just cut the power down.

Re: The impossibility of intelligence explosion

#24

Post by dan_mctree »

pbkhrv wrote: post #16
btilly wrote: post #7

This is a stupid article, and reading it will only make you stupider. First, it is undoubtably true that human brains are better suited to being human than AIs would be. That is irrelevant to the question of whether AIs will be better at the task of creating better technologies. Second, it is absolutely false that improvement is incremental. Consider the game of Go. For a long time, computers sucked at Go. Then we ca…

> If we have a computer program that is able to design better hardware, it can be improved by simply moving it to better hardware. Could you please elaborate? What is it about "better hardware" that makes software that runs on it "better"? Can you define "better"?

'Better' as in achieves more results per unit of time. Which is a fundamental problem for humans too. Many do have the intellectual capacity to invent something like general relativity, but few would be capable of doing so in the very limited timeframe we have available and even fewer actually end up doing that instead of dedicating their thinking time somewhere else. More thinking and more output per timeframe should lead to significant improvements for both human and AI in terms of results, which is generally the meaningful part

Re: The impossibility of intelligence explosion

#25

Post by SilasX »

titzer wrote: post #14

> If intelligence is fundamentally linked to specific sensorimotor modalities, a specific environment, a specific upbringing, and a specific problem to solve, then you cannot hope to arbitrarily increase the intelligence of an agent merely by tuning its brain — no more than you can increase the throughput of a factory line by speeding up the conveyor belt. The latter part makes no sense at all. Of course you can incr…

Also, intelligence includes the ability to identify other bottlenecks and remove them. Of course you can't expect that that a faster conveyor belt, by itself, can yield ever-greater output, but you can expect further improvements if the factory is feeding engineers who are working on ways to further re-design the factory, and they are currently constrained by the factory's output.

The whole reason that some people predict an intelligence explosion is because intelligence is the resource that can find arbitrary ways to self-improve, other than tuning one specific parameter.

I would go so far as to say that intelligence is the same thing as the ability to make improvements that climb out of a local domain of attraction.

Re: The impossibility of intelligence explosion

#26

Post by zentiggr »

btilly wrote: post #7

This is a stupid article, and reading it will only make you stupider. First, it is undoubtably true that human brains are better suited to being human than AIs would be. That is irrelevant to the question of whether AIs will be better at the task of creating better technologies. Second, it is absolutely false that improvement is incremental. Consider the game of Go. For a long time, computers sucked at Go. Then we ca…

> Second, it is absolutely false that improvement is incremental. Consider the game of Go. For a long time, computers sucked at Go. Then we came up with Monte Carlo search about a decade ago and suddenly we were at strong amateur. Then deep learning was applied, and Alpha-Go jumped to able to beat any human in the world, and has continued to improve.

I think you missed his point that this sort of improvement in specific task performance tends to not generalize into improvement at _any_ task. He specifically used AlphaGo as an example of a very narrow specialized problem... i.e, AlphaGo will likely never contribute to solving anything but Go or very closely related tasks.

> Third, linear progress is not the history of technology. The history of technology is exponential progress over and over again on everything from how far a steam ship could travel without refueling to the number of operations per second a CPU can carry out.

Each expansion of which is counterbalanced by many other inefficiencies (maximum speed of steam ship hulls, carrying capacity, heat losses of faster CPUs and timing/latency problems that put limits on signal propagation, etc).

> But on a more relevant level, the task itself is clearly possible. If we have a computer program that is able to design better hardware, it can be improved by simply moving it to better hardware. But it is capable of designing that hardware. This creates a feedback loop which should shorten the cycles in Moore's law, resulting in very few years in a truly superior AI capability.

I think you missed the main argument of the whole article, which was basically that every exponential increase in one measure is rooted in the civilization and circumstances as a whole, and therefore never truly reaches runaway effect. Too many other supporting circumstances are needed for any one change to truly dominate the overall system.

Re: The impossibility of intelligence explosion

#27

Post by titzer »

SilasX wrote: post #25
titzer wrote: post #14

> If intelligence is fundamentally linked to specific sensorimotor modalities, a specific environment, a specific upbringing, and a specific problem to solve, then you cannot hope to arbitrarily increase the intelligence of an agent merely by tuning its brain — no more than you can increase the throughput of a factory line by speeding up the conveyor belt. The latter part makes no sense at all. Of course you can incr…

Also, intelligence includes the ability to identify other bottlenecks and remove them. Of course you can't expect that that a faster conveyor belt, by itself, can yield ever-greater output, but you can expect further improvements if the factory is feeding engineers who are working on ways to further re-design the factory, and they are currently constrained by the factory's output. The whole reason that some people pr…

> The whole reason that some people predict an intelligence explosion is because intelligence is the resource that can find arbitrary ways to self-improve, other than tuning one specific parameter.

Yes, exactly. General intelligence is "thinking about thinking", and that inevitably entails removing bottlenecks in creative ways.

Re: The impossibility of intelligence explosion

#28

Post by breuleux »

btilly wrote: post #7

This is a stupid article, and reading it will only make you stupider. First, it is undoubtably true that human brains are better suited to being human than AIs would be. That is irrelevant to the question of whether AIs will be better at the task of creating better technologies. Second, it is absolutely false that improvement is incremental. Consider the game of Go. For a long time, computers sucked at Go. Then we ca…

> If we have a computer program that is able to design better hardware, it can be improved by simply moving it to better hardware.

This adds a constraint that the new hardware has to be able to run that program, which is an important source of friction in the design. Especially if you consider the case of neuromorphic AI, true improvements in hardware design may involve changes so extensive and so breaking that it wouldn't be possible for the original AI to function on it. In order to exploit the better hardware capabilities, new AI would therefore have to be trained from zero, which is a potentially expensive operation.

So regardless of which way to go, you encounter significant friction: either you constrain your design to support prior software, which is hard, requires a lot of checks, and is severely suboptimal in the grand scheme of things. Or you don't, in which case you have to wait a while until new AI can be made to exploit the new architecture. It's not as simple as you think.

Re: The impossibility of intelligence explosion

#29

Post by btilly »

pbkhrv wrote: post #16
btilly wrote: post #7

This is a stupid article, and reading it will only make you stupider. First, it is undoubtably true that human brains are better suited to being human than AIs would be. That is irrelevant to the question of whether AIs will be better at the task of creating better technologies. Second, it is absolutely false that improvement is incremental. Consider the game of Go. For a long time, computers sucked at Go. Then we ca…

> If we have a computer program that is able to design better hardware, it can be improved by simply moving it to better hardware. Could you please elaborate? What is it about "better hardware" that makes software that runs on it "better"? Can you define "better"?

Hardware that has more memory, more processing speed, faster access to memory, and more parallelism is better than hardware without those characteristics.

The exact same software running on better hardware will run faster and can tackle larger problems.

We can't possibly build a human with twice the memory that thinks twice as fast. However once we have an AI which is roughly equivalent to a human, having an AI with twice as much memory that thinks twice as fast is just 2-5 years. (How long depends on where the bottleneck is.)

Re: The impossibility of intelligence explosion

#30

Post by zentiggr »

btilly wrote: post #8
kfk wrote: post #5

Earlier quoted context omitted.

I think the point he is trying to make is that there are boundaries to intelligence. I think of it this way - no matter how smart an AI is, it still would take 4.3 years to reach Alpha Centauri going at the speed of light . An AI still needs to run experiments, collect evidence, conjure hypothesis, reach consensus, etc.. Is this really that far more efficient than what humans do today?

Is this really that far more efficient than what humans do today? A major managerial problem with humans is sorting out our irrational emotional biases and keeping everyone working on something resembling the appointed task. Can you imagine the productivity gain if that problem suddenly went away?

So emotions are a problem to be solved and moved past...

I don't want your world. Dystopias suck.

Post reply on HN