Imagine we errect ourselves a tomb made of drones to live in like in lems first contact novella. A local conflict spilling into space.. Whos vast distance enforces autonomy. Autonomy which closes the door on space exploration.
A Reply to François Chollet on Intelligence Explosion
81–89 of 89 posts
Re: A Reply to François Chollet on Intelligence Explosion
#82Earlier quoted context omitted.
That makes an assumption that there aren't limits on intelligence regimes and that you can just recursively improve intelligence without much friction. That's a very big assumption that has no basis in evidence.
A cubic foot size system running on 100 watts can implement a human-level intelligence (obviously, we have such a system running in our brains), so the theoretical limit is at least there. It's extremely implausible that a similar system (perhaps requiring 100 times more power and/or space) couldn't implement a human-level intelligence running 100 times faster; doing an hour-and-a-half equivalent of thinking, plannin…
These statements are extremely hand-wavey and provide no evidence for their claims.
Re: A Reply to François Chollet on Intelligence Explosion
#83Why should the "Seed AI" that François speaks of need GPS skills "slightly greater" than that of humans to spark the explosion. Should not just GPS at all do the trick, the way he describes things, since even with human or less than human level skills the computer could still rapidly recurse and self improve?
Re: A Reply to François Chollet on Intelligence Explosion
#84I am not convinced that "super-intelligence" is either possible or a threat.
If it is possible, why are not societies or corporations super-intelligent? I suspect they are not, because they face organizational problems that any system will face. And I don't think these organizational problems can be solved with faster or more communication, but rather that they are fundamental in distributed systems.
But maybe they are (at least in some cases) vastly more intelligent. But then, are they a threat? I think they are more of a threat to themselves than humans..
Re: A Reply to François Chollet on Intelligence Explosion
#85I can't remember where, but I saw a fairly damning argument against the intelligence explosion hypothesis on the grounds that it only works if the algorithm used to design a mind with n units of intelligence (whatever these are) scales linearly with units of intelligence. If it scales faster than linear, then your recursive bootstrapping operation takes longer and longer each time, so that eventually your next bootst…
I think you are basically correct to say that the scalability of AGI is a speculative assumption. The argument is that if it were feasible, it might pose an existential threat, so we should consider it.
If one accepts the plausibility of AGI, then AGI that is bigger and faster than humans does not seem to be much of a stretch, but I certainly cannot imagine what capabilities a qualitatively more powerful intelligence would have, let alone how much effort would be required to get there.
Chollet claims that more powerful intelligence is, in fact, impossible, but as Scott Aaronson pointed out [1], his argument from the No Free Lunch theorem does not have any bearing on the question. As Yudkowsky points out in the article, Chollet's other arguments could just as well be used to claim that nothing beyond the level of intelligence of chimpanzees is possible.
But maybe just a bigger, faster, human-like intelligence might present a risk - people have been outsmarting one another for millennia.
Re: A Reply to François Chollet on Intelligence Explosion
#86Earlier quoted context omitted.
True, but I think that the results of such a "weak intelligence explosion" (where the linear/sublinear scaling case would be a "strong intelligence explosion"), while still remarkable, would fall far short of some of the expectations placed on general AI by the singularity / superintelligence crowd. For the sake of argument, extrapolating from our current energy consumption using a (simplistic) linear model and some…
I don't understand the connection to energy consumption. Humans have been able to extract increasing amounts of energy over time without correspondingly large increases in human intelligence. I don't see a strong reason to doubt that this will continue, so I don't see a strong reason to doubt that a >= human-level-intelligence AI could do it either.
The raw computing power of the human wetware has not increased appreciably over historical timescales, but the total intelligence of humanity, includes, for example, the increase in effective intelligence gained by storing knowledge in external devices like books. The gestalt organism that is "humanity" is much smarter than it was 500 or even 100 years ago, which correlates with our ability to extract resources. It's an extremely simplistic model, but since I was only after a very rough guess, I went with it.
Re: A Reply to François Chollet on Intelligence Explosion
#87I can't remember where, but I saw a fairly damning argument against the intelligence explosion hypothesis on the grounds that it only works if the algorithm used to design a mind with n units of intelligence (whatever these are) scales linearly with units of intelligence. If it scales faster than linear, then your recursive bootstrapping operation takes longer and longer each time, so that eventually your next bootst…
I think you are basically correct to say that the scalability of AGI is a speculative assumption. The argument is that if it were feasible, it might pose an existential threat, so we should consider it. If one accepts the plausibility of AGI, then AGI that is bigger and faster than humans does not seem to be much of a stretch, but I certainly cannot imagine what capabilities a qualitatively more powerful intelligence…
Oh, don't get me wrong, AGI is still terrifying, even if it is limited to a few (or a few hundred) multiples of human intelligence by algorithmic or physical constraints. The discovery of AGI will precipitate the biggest social upheaval we've ever seen. I'm just very skeptical of the quasi-religious side of the superintelligence community, who think that we'll create godminds that can bend reality to their will or something.
Also, it didn't seem to me like Chollet was claiming that better than human intelligence was impossible, honestly. This seems to be a motivated misreading of the original article on Yudkowksy's part, and most of his article is arguing with a straw man because of it. More charitably, Chollet seems to be saying that there may be limitations to intelligence that are "built in" due to the context in which intelligence operates. For instance, even if we create a "superintelligence" in the sense that it has much greater raw processing power than humans, we may not be able to create the sensory environment and training program that would allow it to learn how to recursively improve itself without limit.
Re: A Reply to François Chollet on Intelligence Explosion
#88Earlier quoted context omitted.
I think you are basically correct to say that the scalability of AGI is a speculative assumption. The argument is that if it were feasible, it might pose an existential threat, so we should consider it. If one accepts the plausibility of AGI, then AGI that is bigger and faster than humans does not seem to be much of a stretch, but I certainly cannot imagine what capabilities a qualitatively more powerful intelligence…
Oh, don't get me wrong, AGI is still terrifying, even if it is limited to a few (or a few hundred) multiples of human intelligence by algorithmic or physical constraints. The discovery of AGI will precipitate the biggest social upheaval we've ever seen. I'm just very skeptical of the quasi-religious side of the superintelligence community, who think that we'll create godminds that can bend reality to their will or so…
You are right about Chollet not ruling out growth altogether, but he does say - as a section heading, no less - that "our environment puts a hard limit on our individual intelligence." After a diversion into a non-sequitur about individual humans being incapable of bootstrapping their own intelligence, he launches into an extended argument that intelligence can only grow with the culture it is embedded in, and so intelligence can only grow linearly at best (linearly in what? culture?) The whole argument is a waste of time, because AI apocalypse fears are not predicated on exponential growth (and certainly not growth without limit), but only that it outstrips that of humans (and maybe not even that.) Chollet never seems to address the possibility that AGI might drive its own culture to grow, just as our ancestors' developing intelligence drove the (proto-)human culture to grow. (If Collet were to deny that this happened, then he would not be able to explain why human intelligence outstrips that of other apes, given his position that culture constrains intelligence.)
Re: A Reply to François Chollet on Intelligence Explosion
#89Earlier quoted context omitted.
Firstly, growth rates of non-information tech largely isn't relevant to AI. That said, overall growth in knowledge is at least directly proportional to population growth. Even pessimistically considering humans as dumb, an exponential population growth means an exponential discovery rate just from sheer trial and error. So overall progress is undeniably exponential , even if it's a low exponent. Secondly, we know def…
Overall growth in knowledge isn't at least directly proportional to population growth, if the proportion of knowledge already known and shared grows with population growth. But even if it did, we don't have another doubling of human population ahead of us, so you better hope we're already there. As you point out, Moore's Law doesn't have a ton more power available to it either. Lots of problems don't parallelize well…
> Overall growth in knowledge isn't at least directly proportional to population growth, if the proportion of knowledge already known and shared grows with population growth.
I don't see why.
> But even if it did, we don't have another doubling of human population ahead of us, so you better hope we're already there
Right, but we have plenty of doublings of intelligent non-human agents ahead of us. Until then, we increase our effective intelligence using semi-intelligent machines, like we've been augmenting our physical strength with mechanical devices for millennia.
> As you point out, Moore's Law doesn't have a ton more power available to it either.
I disagree. Frequency scaling won't yield too much more improvement. There are other scaling modes available though, as I described. Moore's law is about information density, not performance.
> Lots of problems don't parallelize well
Often repeated, but frequently overstated. Our knowledge of parallelism is still in its infancy.
> quantum computing has never demonstrated more power than classical computing,
Any other option would require rewriting a lot of physics.
> hardware growth doesn't in fact guarantee an intelligence explosion.
Increased information density beyond that available in the human brain means simulating said brain is feasible. That's as close to a guarantee as you can get.
> Look, every futurist in the world in 1975 thought that by 2017, we'd all be routinely traveling faster than sound, that we'd have colonies on the moon if not mars, and that probably we'd have AGI or something pretty close to it by 2017
Except I'm not giving a timeline, I'm saying it's inevitable. Low exponent exponential growth is still exponential. The intelligence explosion is about a trend, not a fixed milestone.