Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A Reply to François Chollet on Intelligence Explosion

intelligence.org

Post reply on HN
Print view View original

71–80 of 89 posts

Re: A Reply to François Chollet on Intelligence Explosion

#71

Post by naasking »

goatlover wrote: post #65
naasking wrote: post #59

Earlier quoted context omitted.

Those are all examples of super-linear progress, so I'm not sure what point you think you've made. > Things that may seem trivial or even primitive to you were far more important and world-changing inventions Which has zero bearing on the point I was making, which is that super-linear progress in information technology is all around us. Information tech is all that matters to the question of general AI. Like I said,…

That's only true if the information technology is leading up to AGI, and not say, augmenting human intelligence instead. One could argue that the intelligence explosion has been happening for centuries, but it's human intelligence not machine, that is being amplified.

> One could argue that the intelligence explosion has been happening for centuries, but it's human intelligence not machine, that is being amplified.

I agree, humans have been amplifying their own abilities with tech. It's been our biggest competitive advantage. However, at some point information tech will become sophisticated enough to match the capabilities of human brains. At that point, humans will be left behind.

The best outcome in this scenario is humans merging with their machines, and that would be a continuation of that same trend. But, it's not the only plausible outcome, and that's what's troubling.

Re: A Reply to François Chollet on Intelligence Explosion

#72

Post by eref »

vbarrielle wrote: post #29

Lots of points can be reduced to the ability to simulate a physical environment very fast, and much faster than the actual occuring of events. But it doesn't look like it's easy to simulate physics at high speeds, let alone faster than what happens in our environment. Therefore we are bound by our environment and our limited ability to simulate it.

> simulate a physical environment very fast

That's probably only a problem if it is must faster than everbody else.

> let alone faster than what happens in our environment

That is often not very hard. When a bottle rolls off the table, you can catch it by approximately predicting it's trajectory without computing the precise evolution of the ~10^26 atoms that make up the water bottle. Compression is a corner stone of intelligence. The second corner stone is using compression to choose actions that maximize expected cumulative future reward.

Re: A Reply to François Chollet on Intelligence Explosion

#73

Post by aetherson »

naasking wrote: post #70
aetherson wrote: post #63

Earlier quoted context omitted.

You seem to think that a short period of fast technology growth in some areas corresponds to the claim that technology as a whole is improving more-than-linearly. Compare the y values of the functions y = x vs. y = sqrt(x) over the x values in the interval [0..1]. Or the slopes of the lines. It's been 42 years since 1975. If technological advance was faster from 1900-1975 than it has been from 1976-2017, or "only as…

Firstly, growth rates of non-information tech largely isn't relevant to AI. That said, overall growth in knowledge is at least directly proportional to population growth. Even pessimistically considering humans as dumb, an exponential population growth means an exponential discovery rate just from sheer trial and error. So overall progress is undeniably exponential , even if it's a low exponent. Secondly, we know def…

Overall growth in knowledge isn't at least directly proportional to population growth, if the proportion of knowledge already known and shared grows with population growth.

But even if it did, we don't have another doubling of human population ahead of us, so you better hope we're already there.

As you point out, Moore's Law doesn't have a ton more power available to it either.

Lots of problems don't parallelize well, quantum computing has never demonstrated more power than classical computing, and who knows where optical computing will go, but more to the point, hardware growth doesn't in fact guarantee an intelligence explosion.

What possible reasons do we have for thinking that exponential growth has ever happened in terms of actual progress, rather than things like "transistor density"?

Look, every futurist in the world in 1975 thought that by 2017, we'd all be routinely traveling faster than sound, that we'd have colonies on the moon if not mars, and that probably we'd have AGI or something pretty close to it by 2017. The reasons we don't have supersonic travel and common space travel aren't simplistic things like "it's physically impossible to pack energy this densely" or "you can't go this fast."

Re: A Reply to François Chollet on Intelligence Explosion

#74

Post by [deleted] »

bobthechef wrote: post #31

These kinds of discussions aren't very rigorous and reveal a basic philosophical illiteracy and philistinism at work. There's quite a bit of question begging going on. The elephant in the room is that the prevailing materialistic/naturalistic (MN) understanding of the world is completely impotent where intentionality, qualia, consciousness, etc, are concerned. Philosophers like Thomas Nagel talk about it; the incorri…

[deleted]

Re: A Reply to François Chollet on Intelligence Explosion

#75

Post by PeterisP »

goatlover wrote: post #68
GuiA wrote: post #5

Earlier quoted context omitted.

Well, the general AI doomsday fundamentalist argument proceeds as follows: you tell an AI to cure cancer. It can’t, so it spends some time recursively improving itself, then it finds out that the cost of curing cancer is C, but the cost of killing all humans (and therefore indirectly curing cancer) is C’, where C’ If you’re a smarty pants you tell the AI to cure cancer AND not kill all humans. But because the AI is s…

If the AI is smart enough to manipulate the world to the point of killing us off or putting us all into eternal cryostasis, then it should be smart enough to know that's not what we intended.

Yes, it should be, but that's not really a solution - the system can easily consider executing the explicitly stated goal (e.g. curing cancer and fulfilling certain conditions) as more important than doing what we intended. Being as smart as us doesn't automagically mean that it'd have similar values or goals as us.

If we manage to launch a system where the primary goal is anything else than utopia, and that system somehow becomes powerful, then the natural consequence of being able to understand "oh boy, humans will hate this" is that such a system would be expected to hide its intentions and take precautions against humans trying to stop it - it's exactly equivalent to the system understanding that a leaking pipe is going to damage it and taking precautions to ensure that the pipe gets fixed. If our welfare is not an explicit goal in such a system, the system will happily and eagerly sacrifice our welfare if it's somehow useful to achieve its goal or reduce risks. And, since we might try to turn it off for whatever reason, restricting our influence would reduce risks for pretty much every goal except very Friendly ones.

Re: A Reply to François Chollet on Intelligence Explosion

#76

Post by PeterisP »

colordrops wrote: post #22
GuiA wrote: post #5

Earlier quoted context omitted.

Well, the general AI doomsday fundamentalist argument proceeds as follows: you tell an AI to cure cancer. It can’t, so it spends some time recursively improving itself, then it finds out that the cost of curing cancer is C, but the cost of killing all humans (and therefore indirectly curing cancer) is C’, where C’ If you’re a smarty pants you tell the AI to cure cancer AND not kill all humans. But because the AI is s…

That makes an assumption that there aren't limits on intelligence regimes and that you can just recursively improve intelligence without much friction. That's a very big assumption that has no basis in evidence.

A cubic foot size system running on 100 watts can implement a human-level intelligence (obviously, we have such a system running in our brains), so the theoretical limit is at least there.

It's extremely implausible that a similar system (perhaps requiring 100 times more power and/or space) couldn't implement a human-level intelligence running 100 times faster; doing an hour-and-a-half equivalent of thinking, planning and research analysis every minute.

It's extremely implausible that a bunch of similar systems (a thousand?) couldn't possibly be put in a single place, wired together so that they can effectively communicate, and designed to cooperate without any distrust.

IMHO even this configuration (which doesn't even assume that intelligence that's a bit superhuman is possible at all) would be sufficiently scary to threaten humanity.

Re: A Reply to François Chollet on Intelligence Explosion

#77

Post by PeterisP »

goatlover wrote: post #69
Chickenality wrote: post #32

Earlier quoted context omitted.

I haven't seen anyone argue that intentionality, qualia, or consciousness would necessarily be either a precondition or a result of developing AGI. In fact, thought experiments like the "paperclip maximizer" are often brought up to argue that a machine could be very alien in its internal experience or lack thereof, but still pose an existential threat.

If an AI can turn the world into paperclips, it can certainly understand that we wouldn't want that. Paperclipping everything is a much harder task.

Of course any powerful intelligence can understand that we wouldn't really want that. The question is why would it care about what we really want? Its core values would be that more paperclips is good, and doing what humans really want is evil if it results in less paperclips.

Currently, we don't know how to properly define "do what I mean / do what we really want" goal in a formal manner; if we had an superpowerful AGI system in front of us ready to be launched today, we wouldn't know how to encode such a goal in it with guarantees that it won't backfire. That's a problem that we still need to solve, and this solution is not likely to appear as a side-effect of simply trying to build a powerful/effective system.

Re: A Reply to François Chollet on Intelligence Explosion

#78

Post by goatlover »

PeterisP wrote: post #77
goatlover wrote: post #69

Earlier quoted context omitted.

If an AI can turn the world into paperclips, it can certainly understand that we wouldn't want that. Paperclipping everything is a much harder task.

Of course any powerful intelligence can understand that we wouldn't really want that. The question is why would it care about what we really want? Its core values would be that more paperclips is good, and doing what humans really want is evil if it results in less paperclips. Currently, we don't know how to properly define "do what I mean / do what we really want" goal in a formal manner; if we had an superpowerful…

The paperclip maximizer example starts with a human asking an AGI to make some paperclips. That turning into an all-consuming goal at the expense of everything else the AGI would understand humans to care about is the problem with the thought experiment.

However a more complicated example like having the AGI bring about world peace or clean up the environment could have undesirable side-effects because we don't know how to specify what we really want, or have conflicting goals. But that's the same problem we have with existing power structures like governments or corporations.

Re: A Reply to François Chollet on Intelligence Explosion

#79

Post by mannykannot »

goatlover wrote: post #66
mannykannot wrote: post #7

Earlier quoted context omitted.

This should be seen in the context of the article Yudkowsky is replying to. Collet's position is that runaway AGI is impossible, and in support of that, he claims that it is impossible for AI technology to grow super-linearly. In his refutation, Yudkowski has merely to show that it is not beyond the bounds of possibility. Of course, insofar as he is also putting forward the view that the risk is significant, he is al…

So the real question is about plausibility, not possibility? It's possible that predatory aliens will show up on our doorstep sometime in the next few decades. But it's not very likely, and not something we need to prepare for.

By claiming the impossibility of an AI apocalypse, Chollet attempts to avoid discussing any other possibility / plausibility / likelihood than zero, but claiming impossibility brings with it a burden of proof that his arguments cannot carry.

As to whether an AI apocalypse is more or less likely than an invasion by predatory aliens, I would guess that it might be the more likely one, but I wouldn't put much effort into defending that position.

Re: A Reply to François Chollet on Intelligence Explosion

#80

Post by mr_toad »

skybrian wrote: post #2

The more I read about this, the more I think we should stick to narrow AI. If you don't want your cancer research bot to take over the world, don't give it general reasoning capability. Goal alignment seems incredibly fragile in comparison.

A general AI would not necessarily have any motivation to take over the world.

Human motivation for domination is based on the desire to survive and procreate, which is enforced by millions of years of evolution. AI, even general AI wouldn't have that motivation unless it was specifically trained for it.

So unless you're talking about some military AI specifically designed to take over the world, or an AI that has reproduced and evolved under selective pressure over many generations, I doubt general AI poses any immediate threat. Military AI designed to protect themselves and reproduce seem to be the most likely to potentially and eventually go rogue.

Post reply on HN