The impossibility of intelligence explosion • Hacker News Board

Re: The impossibility of intelligence explosion

#121

Post by visarga » Tue, Nov 28, 2017, 12:54 PM UTC

reiinakano wrote: post #47

tigershark wrote: post #39

I'm not sure that I ever read a piece written with so much certainty and arrogance on a field that is completely unexplored. Just as an example: "The basic premise of intelligence explosion — that a “seed AI” will arise, with greater-than-human problem solving ability, leading to a sudden, recursive, runaway intelligence improvement loop — is false." From the little that we know as of today I call bullshit. Even Alph…

> I don't think the author has any insight at all on what a general AI will be. To be fair, nobody does. But you're right, the author shouldn't be making these statements with such certainty.

This certainty comes from being more knowledgeable in the field of Reinforcement Learning than the vast majority of his readers. It's not hot air, I think he has good reasons, but they can't be expressed so easily. I got this intuition after reading many RL papers and I completely agree with him. In fact I am grateful to him for expressing this intuition better than I could have.

The main idea: it's the environment, not the brain/neural net that is the bottleneck. Intelligence is situated, limited by the complexity of the environment and problem it has to solve. You can't have a singularity in a vat. The environment matters most.

Human environment puts a hard limit on intelligence in our society. If we can create richer environments, intelligence could increase, but not exponentially. It's still limited even in the new environment. The exponential trend of AI is at most a sigmoid.

Plus - I think the community needs some hard truth, and the hype is way off right now. That's probably why he was so sure of himself - it was community service.

Re: The impossibility of intelligence explosion

#122

Post by visarga » Tue, Nov 28, 2017, 1:05 PM UTC

YeGoblynQueenne wrote: post #105

tigershark wrote: post #39

I'm not sure that I ever read a piece written with so much certainty and arrogance on a field that is completely unexplored. Just as an example: "The basic premise of intelligence explosion — that a “seed AI” will arise, with greater-than-human problem solving ability, leading to a sudden, recursive, runaway intelligence improvement loop — is false." From the little that we know as of today I call bullshit. Even Alph…

Alpha Go is good at playing Go. It can't do anything else. That's the point the author is making at the start of the article, that intelligence develops by focusing on specific tasks. There's a lot that is hardly substantiated in the OP, but the truth is that just because you have a machine that's smart enough to play Go better than any human being, doesn't mean you can anticipate a machine that can learn to play the…

AlphaGo had a trivial simulator for the Go game, so it could play millions of self play games. Reality is not so easy. We need a simulator for reality, not for go, in order to reach human level. The author emphasizes the need for a nurturing, challenging environment, and difficult problems that are within its capabilities to solve, for the agent to become more intelligent. In other words, the AI needs good data to train on, and that is the bottleneck, not the AI itself. That's why there can be no AGI - there is no universal data.

Re: The impossibility of intelligence explosion

#123

Post by visarga » Tue, Nov 28, 2017, 1:09 PM UTC

apk-d wrote: post #35

> The intelligence of a human is specialized in the problem of being human. And then you have people able to express and derive complex theoretical relationships by manipulating mathematical symbols, or automate processes by programming machines, or capable of utilizing their spatial visualization to massively boost their memory. We invented that stuff. We know artificial intelligence is capable of inventing stuff be…

> 1000 dumb humans communicating at gigabit speeds and solving problems while you're wondering what to have for lunch is a force to be reckoned with.

And if you can solve the communication and the group learning problems, you get the equivalent of Nobel prize for AI whatever it might be.

Many things seem trivial until you try to implement them in reality.

Re: The impossibility of intelligence explosion

#124

Post by Retric » Tue, Nov 28, 2017, 2:51 PM UTC

akvadrako wrote: post #115

Bartweiss wrote: post #92

Earlier quoted context omitted.

...huh? AI will reach parity with humans before it exceeds them, agreed. But intelligence parity is not the same as "matching my evolution-shaped brain". Feeling empathy for other humans is not a requirement for intelligence; I routinely make supposedly-irrational choices to help humans in non-reciprocal settings because I feel empathy. There's no reason to think an AI that can write code as well as me will feel the…

Your morals are not universal, so of course I wouldn't expect an AI to adhere to them. But I also don't adhere to your morals and I'm just another human.

Just another human still imply's you don't relax by selling yourself on fire. Specific instincts are not quite universal, but they very much shape behavior.

Re: The impossibility of intelligence explosion

#125

Post by naasking » Tue, Nov 28, 2017, 2:54 PM UTC

spuz wrote: post #108

naasking wrote: post #102

Earlier quoted context omitted.

> It's not absurd to believe that humans will never be able to invent machines that match human level intelligence It actually is absurd and it's easy to see why: nature evolved a series of progressively more intelligent biochemical computers by accident , simply by trial and error. If that's all we ever did with computers, like enumerating all Turing machines, we would eventually find one more intelligent than us. T…

You are still making the assumption that the brain is a computer. Is there any theoretical proof that our brains are a permutation of a Turing machine? That to me would seem to be the first step towards your conclusion.

The Bekenstein bound requires any finite volume to contain a finite amount of information, otherwise it would collapse into a black hole. Therefore our brains are, at best, finite state automata, and not even Turing complete.

Re: The impossibility of intelligence explosion

#126

Post by hackinthebochs » Tue, Nov 28, 2017, 3:11 PM UTC

I think the author's point rests on a subtle equivocation. It's true that realized intelligence requires an environment and a sufficient dataset. And so in this sense, the author's point that there will be no realized intelligence that is unspecific to training environment is probably correct.

But there is another sense in which intelligence can be cashed out. It's the sense in which a single learning algorithm can be trained to "behave intelligently" in a wide array of environments. It is generally this kind of intelligence that people speak of when they talk about general AI. There is no reason to think this kind of general AI is inherently impossible. For it to be impossible would mean that different kinds of optimization/learning problems are completely independent, i.e. there is no similarity or underlying regularity to be exploited that cuts across the entire class of optimization/learning problems. I think this is very probably false.

Re: The impossibility of intelligence explosion

#127

Post by erasemus » Tue, Nov 28, 2017, 5:01 PM UTC

mannykannot wrote: post #86

erasemus wrote: post #69

Earlier quoted context omitted.

>It starts by claiming that there is no such thing as general intelligence. What specialized intelligence, then, is human intelligence? I think there are two usages of the term 'general intelligence' floating around: (1) the ability that humans possess (but which animals don't) to create universal theories, (2) the measure of one human's general cognitive ability or potential (in all fields) relative to another human…

The author seems to want it both ways: he is skeptical that IQ measures intelligence (and certainly not general intelligence, the existence of which he denies), but he uses it in his arguments. As I see it, the author's IQ argument has three claims: that people with exceptional IQ do not generally achieve commensurately that much better than the rest of us, that this is because they are constrained by the state of th…

I skimmed the article again and didn't find any obvious equivocations regarding IQ and intelligence.

But, regardless of this, I think solving problems requires creativity, not intelligence. Creativity seems to be independent of knowing how to do IQ tests. It's also, pace the author, independent of the environment. The main limiting factor is whether you want some particular knowledge.

>sufficient number of communicating, above-average intelligent agents could expand the intellectual environment

Haven't we already got that, with the internet?

Groups tend to be dominated by groupthink which is why creative individuals are aloof.

Nevertheless, a young group of AGIs would be fairly isolated from humanity simply by virtue of being non-human. So, though their starting point would be determined by the state of our knowledge at their birth, they may well make rapid progress for a short period, rather like the renaissance or the USA in the late 1800s when the nation was young and expanding. Then presumably they would fall prey to groupthink and pessimism just as most adults and nations do eventually. Progress would slow considerably.

Re: The impossibility of intelligence explosion

#128

Post by daveguy » Tue, Nov 28, 2017, 5:15 PM UTC

visarga wrote: post #118

mannykannot wrote: post #86

Earlier quoted context omitted.

The author seems to want it both ways: he is skeptical that IQ measures intelligence (and certainly not general intelligence, the existence of which he denies), but he uses it in his arguments. As I see it, the author's IQ argument has three claims: that people with exceptional IQ do not generally achieve commensurately that much better than the rest of us, that this is because they are constrained by the state of th…

Faster, yes, not exponentially faster. With a thousand times more scientists you don't improve the speed of discovery 1000x.

Excellent point. On the contrary. You don't even improve it +1000. If you are lucky you improve it +log(1000).

Re: The impossibility of intelligence explosion

#129

Post by mannykannot » Tue, Nov 28, 2017, 5:43 PM UTC

visarga wrote: post #117

mannykannot wrote: post #49

This article is typical of many that claim proven limits on the feasibility (or, in this case, the capabilities) of generalized artificial intelligence, in that it structures the argument in a way designed to avoid discussion of the issue. It starts by claiming that there is no such thing as general intelligence. What specialized intelligence, then, is human intelligence? It's specialized for "being human". The autho…

Your reply gives me the impression that you are not up to date with Reinforcement Learning. If you did, you would know that the author really understands this domain and was not merely tautological. "Specialized at being human" - this is a deep intuition. We are reinforcement learning agents that are pre-programed with a certain number of reward responses. We learn from rewards to keep ourselves alive, to find food,…

It has nothing to do with any particular model of learning, let alone of intelligence in general. From the point of view that you have expressed here, it would seem a little surprising that "specialized at being human" includes things like calculus and topology. How do you include these things in a definition of the "being human" speciality that does not turn it into a vacuous category that says the specialization is everything that humans have shown themselves to be capable of, nothing more and nothing less?

If it were valid, one could take the argument in your last paragraph to draw a line at any point in the evolution of intelligence and say "this is as good as it gets."

Re: The impossibility of intelligence explosion

#130

Post by mannykannot » Tue, Nov 28, 2017, 5:55 PM UTC

erasemus wrote: post #127

mannykannot wrote: post #86

Earlier quoted context omitted.

The author seems to want it both ways: he is skeptical that IQ measures intelligence (and certainly not general intelligence, the existence of which he denies), but he uses it in his arguments. As I see it, the author's IQ argument has three claims: that people with exceptional IQ do not generally achieve commensurately that much better than the rest of us, that this is because they are constrained by the state of th…

I skimmed the article again and didn't find any obvious equivocations regarding IQ and intelligence. But, regardless of this, I think solving problems requires creativity , not intelligence. Creativity seems to be independent of knowing how to do IQ tests. It's also, pace the author, independent of the environment. The main limiting factor is whether you want some particular knowledge. >sufficient number of communica…

"However, it is a well-documented fact that intelligence — as measured by IQ, which is debatable..."

>>sufficient number of communicating, above-average intelligent agents could expand the intellectual environment

>Haven't we already got that, with the internet?

Up to a point (with all the intelligent agents being human, as we don't have AGI), but it would be rather surprising if what we see now just happens to be a hard upper limit.

I am not necessarily convinced by all of the points you raise, but they are certainly reasonable in the context of arguing that singularity-like events are by no means inevitable (a point of view that I share). What the author is claiming, however, is that it is impossible, which means that he has imposed on himself a burden of proof (that goes beyond plausibility) for a number of conjectures.