Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A Reply to François Chollet on Intelligence Explosion

intelligence.org

Post reply on HN
Print view View original

41–50 of 89 posts

Re: A Reply to François Chollet on Intelligence Explosion

#41

Post by evanwise »

mundo wrote: post #19
pasquinelli wrote: post #4

Earlier quoted context omitted.

by what means will a cancer research bot take over the world? it doesn't matter how smart you are if you don't have the necessary means to do something. i think it's a fantasy, something that people who imagine themselves to be very intelligent have latched onto-- the idea that their best quality is the best quality.

a) If it can't surprise you, it's not really intelligent b) If it can surprise you, it can do so negatively That's all you need to demonstrate that the danger exists , that an AI can mis-use the tools you give to it. The simplicity of it makes it pretty irrefutable. Separate from that, the extent of the danger depends entirely on the details of what the AI does and what it's hooked up. Sure, an AI that can't do anyth…

A lot of folks who are critical of the AI safety movement also miss the potential for "side channel attacks" where the AI learns to manipulate human actors and trick them into "unbottling the genie". Honestly, this seems a lot more plausible to me than most AI disaster scenarios.

Re: A Reply to François Chollet on Intelligence Explosion

#42

Post by Chickenality »

evanwise wrote: post #39
rhaps0dy wrote: post #35

Earlier quoted context omitted.

>meaning there is some finite intelligence cap for any such bootstrapped mind This does not preclude an intelligence explosion, this cap could be many (say, 100) times higher than human intelligence. We could still see many features of an explosion in that case.

True, but I think that the results of such a "weak intelligence explosion" (where the linear/sublinear scaling case would be a "strong intelligence explosion"), while still remarkable, would fall far short of some of the expectations placed on general AI by the singularity / superintelligence crowd. For the sake of argument, extrapolating from our current energy consumption using a (simplistic) linear model and some…

I don't understand the connection to energy consumption. Humans have been able to extract increasing amounts of energy over time without correspondingly large increases in human intelligence. I don't see a strong reason to doubt that this will continue, so I don't see a strong reason to doubt that a >= human-level-intelligence AI could do it either.

Re: A Reply to François Chollet on Intelligence Explosion

#43

Post by jtolmar »

evanwise wrote: post #39
rhaps0dy wrote: post #35

Earlier quoted context omitted.

>meaning there is some finite intelligence cap for any such bootstrapped mind This does not preclude an intelligence explosion, this cap could be many (say, 100) times higher than human intelligence. We could still see many features of an explosion in that case.

True, but I think that the results of such a "weak intelligence explosion" (where the linear/sublinear scaling case would be a "strong intelligence explosion"), while still remarkable, would fall far short of some of the expectations placed on general AI by the singularity / superintelligence crowd. For the sake of argument, extrapolating from our current energy consumption using a (simplistic) linear model and some…

That's a very strange extrapolation - we don't need to be any smarter to extract more energy. We're building more renewable energy capability every year, even though humans aren't getting any smarter.

Building a dyson swarm would be a massive project, but modern humans are plenty smart enough to do it. An AI capable of running the project, while beyond the current state of the art, would still not need to be particularly clever. (It doesn't need to design satellites or space factories to get there.)

Re: A Reply to François Chollet on Intelligence Explosion

#44

Post by pveierland »

vbarrielle wrote: post #29

Lots of points can be reduced to the ability to simulate a physical environment very fast, and much faster than the actual occuring of events. But it doesn't look like it's easy to simulate physics at high speeds, let alone faster than what happens in our environment. Therefore we are bound by our environment and our limited ability to simulate it.

One big advantage with machine intelligence is that you can train many agents simultaneously in many environments, letting the agents share what they learn. Even if you were required to train the agents outside simulated environments, their shared learning would allow for more rapid collective progress.

Re: A Reply to François Chollet on Intelligence Explosion

#45

Post by vbarrielle »

Chickenality wrote: post #36
vbarrielle wrote: post #29

Lots of points can be reduced to the ability to simulate a physical environment very fast, and much faster than the actual occuring of events. But it doesn't look like it's easy to simulate physics at high speeds, let alone faster than what happens in our environment. Therefore we are bound by our environment and our limited ability to simulate it.

This is probably a good criticism if it turns out that the right level of abstraction for most problems is Physics. And it seems like your argument would apply equally well against the idea of human intelligence. Luckily, our minds have developed other abstractions that allow us to solve problems much faster than if we had to simulate them as physics problems. For example, I don't need a physics-level simulation of m…

You're right, there are lots of problems where a simpler abstraction is possible.

But I don't think my argument applies to human intelligence, it just means that human intelligence is what you can get with all the data points you can get by observing the world (and some simulation done by our brains, but I'm under the impression that our brains don't perform accurate simulation, looks more like heuristics).

Re: A Reply to François Chollet on Intelligence Explosion

#46

Post by jjaredsimpson »

dekhn wrote: post #38
cdancette wrote: post #24

Earlier quoted context omitted.

for all functions, f(x) = f(0) + f'(0) * x for x very close to 0

yeah I didn't know that part, just the more specific one.

It's the first term of the Maclaurin series

sin(x) = sin(0) + cos(1)x = 0 + x

Re: A Reply to François Chollet on Intelligence Explosion

#47

Post by cbennett »

evanwise wrote: post #33

I can't remember where, but I saw a fairly damning argument against the intelligence explosion hypothesis on the grounds that it only works if the algorithm used to design a mind with n units of intelligence (whatever these are) scales linearly with units of intelligence. If it scales faster than linear, then your recursive bootstrapping operation takes longer and longer each time, so that eventually your next bootst…

> If it scales faster than linear, then your recursive bootstrapping operation takes longer and longer each time,

Wait, does that really follow? What if you have a better than linear bootstrapping compiler. To unpack that a bit, imagine we not only have $n$ such units, but we have them wired to together in a creative way-- i dont know whether it is hierarchy, or some clever topology, but lets say that the bootstrapper now gets sub-linear scaling properties as it grows $n$.

If we look at the brain, there is a lot to be understood from the dynamics of recurrent neural fields. They are wired in a very complex way which seems to allow for some kind of very special booting (re-booting) operations. And thats just at one level of abstraction, then we re-wire them into meta-fields (like the columnar abstractions that Hawkin's builds his HTM theories around). If we have a sort of fractal information encoding, we ultimately approach shannon efficient coding. Is that what evolution has selected brains to do? And do you think it is possible the first seed AI may realize this and exploit the same strategy, just 1000x (10kx?) faster?

Re: A Reply to François Chollet on Intelligence Explosion

#49

Post by btilly »

aetherson wrote: post #3

Something to note about the "we're on an exponential (or at least more-than-linear) increase in technology" arguments: They generally need to reach back quite a long time to sound convincing. Yudkowsky makes two arguments about how fast technology or society is evolving: in one he chooses 500 years ago, 1517. In another, he talks about "the last 10,000 years." In contrast, Chollet compares 1900-1950 with 1950-2000. I…

In The Innovator's Dilemma over a dozen technologies were documented that were on exponential improvement curves for a period of decades or more. They generally last until the technology either hits a wall in physical possibility, or the thing being delivered is no longer the primary measure of the technology.

Here are some examples off of the top of my head that are happening right now:

Operations per second of a CPU (The famous Moore's law) bits stored per dollar of RAM Energy density of batteries Energy produced per dollar of solar panels

Here are some examples from the past.

Distance a steam ship could travel without refueling Maximum power of a gasoline engine Volume of dirt a hydraulic scoop can pick up

The history of technological progress is dominated by exponential curves. Saying that it is logically impossible for the future to be likewise dominated by exponential curves is just silly.

Re: A Reply to François Chollet on Intelligence Explosion

#50

Post by mannykannot »

bobthechef wrote: post #31

These kinds of discussions aren't very rigorous and reveal a basic philosophical illiteracy and philistinism at work. There's quite a bit of question begging going on. The elephant in the room is that the prevailing materialistic/naturalistic (MN) understanding of the world is completely impotent where intentionality, qualia, consciousness, etc, are concerned. Philosophers like Thomas Nagel talk about it; the incorri…

Putting aside the question of whether qualia arguments are ultimately based on appeals to plausibility, it does not matter in this case, because the behavior of an artificial p-zombie (APZ?) would be indistinguishable from a conscious AGI, and therefore potentially present as much of an existential threat as the latter.
Post reply on HN