Is Intelligence Self-Limiting?

#1

Post by lavin » Sat, Mar 10, 2012, 2:35 PM UTC

"Mobile AI robots do all the work autonomously. If they are damaged, they are smart enough to get themselves to a repair shop, where they have complete access to a parts fabricator and all of their internal designs. They can also upgrade their hardware and programming to incorporate new designs, which they can create themselves."

Okay it's too soon to talk about self-programmed robots. But I think it could be interesting to think about easier self-programmed things. They may not be too far away from now.

Is Intelligence Self-Limiting?
ieet.org

Re: Is Intelligence Self-Limiting?

#2

Post by carsongross » Sat, Mar 10, 2012, 3:26 PM UTC

Reality is self-limiting.

There are no exponentials, only sigmoid curves.

Unfortunately there is a lot of money to be made convincing people that an early stage sigmoid is actually an exponential.

Re: Is Intelligence Self-Limiting?

#3

Post by cousin_it » Sat, Mar 10, 2012, 3:40 PM UTC

Not all possible AI designs are based on reinforcement learning or reward channels. You can code the AI with a formally specified goal instead. For example, "is there a proof of the Riemann hypothesis in ZFC shorter than a million symbols?" If such a goal requires converting the solar system into computronium, then the AI will do that. It won't settle for wireheading because wireheading doesn't give an answer to the formal problem.

To everyone who thinks intelligence might be limited in principle: there's no reason to think humans are anywhere close to the upper limit. In fact there's ample reason to think that humans are at the lowest threshold of intelligence that makes a technological civilization possible, because if we'd reached that threshold earlier in our evolution, we'd have created civilization then instead of now. There's probably plenty of room above us.

Re: Is Intelligence Self-Limiting?

#4

Post by robertskmiles » Sat, Mar 10, 2012, 3:50 PM UTC

carsongross wrote: post #2

Reality is self-limiting. There are no exponentials, only sigmoid curves. Unfortunately there is a lot of money to be made convincing people that an early stage sigmoid is actually an exponential.

Obviously intelligence tops out at the high end of a sigmoid curve - there's only so much matter and energy in the universe to convert to computronium - but that's no reason to suspect any value smaller than that as the limit. It must eventually level off, but the point at which it levels off may be hundreds of billions of times more intelligent than a human.

So the fact that intelligence is limited, doesn't in any way mean that hyperintelligence isn't possible. There is a limit, but we have no reason to believe that limit is on anything like the same order of magnitude as current intelligence.

Re: Is Intelligence Self-Limiting?

#5

Post by [deleted] » Sat, Mar 10, 2012, 3:52 PM UTC

[deleted]

Re: Is Intelligence Self-Limiting?

#6

Post by robertskmiles » Sat, Mar 10, 2012, 3:57 PM UTC

What's the purpose of this 'pleasure' construct in the AI's mind? If it's able to calculate the value of a utility function in order to set the 'pleasure' variable, and it bases its actions on the value of this 'pleasure' variable, why not just cut out the middle man and have it base its actions directly on the result of the utility function? The variable functionally buys you nothing, but introduces this problem that adjusting the variable directly can cause the AI to take actions inconsistent with its utility function.

Without the variable, the problem doesn't happen. The AI values collecting ore. If it has enough self-awareness to reliably modify itself, it knows that if it modifies its utility function it is liable to collect less ore, which is something it doesn't want. The action of modifying the utility function naturally rates very low on the utility function itself.

You don't want to murder people, so not only do you choose not to murder people, but if you are presented with a pill which will make you think it's good to murder people and take great joy in it, you will choose not to take that pill. No matter how enjoyable and good murder may be for you if you take the pill, your own self-knowledge and current utility function prohibit taking it.

The model of intelligence described can be thought of as self-limiting. Luckily it is not by any means the only viable model of intelligence.

Re: Is Intelligence Self-Limiting?

#7

Post by ihnorton » Sat, Mar 10, 2012, 4:05 PM UTC

cousin_it wrote: post #3

Not all possible AI designs are based on reinforcement learning or reward channels. You can code the AI with a formally specified goal instead. For example, "is there a proof of the Riemann hypothesis in ZFC shorter than a million symbols?" If such a goal requires converting the solar system into computronium, then the AI will do that. It won't settle for wireheading because wireheading doesn't give an answer to the…

I think there is a necessary discontinuity between a self-modifying solver algorithm that acts within an internally-consistent formal framework; and an intelligence that is able to modify the formal system itself. On some level this might be considered analogous to self-awareness. Past that discontinuity though, my guess is that all bets are off. There are many historical examples of the depths of madness that may be plumbed in unfulfilled pursuit of a formal goal.

To your second point, absolutely agreed: http://infoproc.blogspot.com/2012/03/only-he-was-fully-awake...

Re: Is Intelligence Self-Limiting?

#8

Post by Swizec » Sat, Mar 10, 2012, 4:22 PM UTC

Valid argument, but we have a clear example that it is wrong.

Humans.

A lot of what we do is driven by internal value calculations and pleasure centers, so why aren't we all simply taking drugs and avoiding all this messy "doing things" business?

Point is, if humans figured out a way of avoiding purely pressing the right buttons to enjoy themselves and actually being useful, so too will smart robots.

Re: Is Intelligence Self-Limiting?

#9

Post by ams6110 » Sat, Mar 10, 2012, 4:30 PM UTC

Swizec wrote: post #8

Valid argument, but we have a clear example that it is wrong. Humans. A lot of what we do is driven by internal value calculations and pleasure centers, so why aren't we all simply taking drugs and avoiding all this messy "doing things" business? Point is, if humans figured out a way of avoiding purely pressing the right buttons to enjoy themselves and actually being useful, so too will smart robots.

why aren't we all simply taking drugs and avoiding all this messy "doing things" business

Some of us are, but that is also self-limiting. If too many of us did it we'd start dying out.

Re: Is Intelligence Self-Limiting?

#10

Post by goodside » Sat, Mar 10, 2012, 4:36 PM UTC

The fact the author can think of one particular way to design an AI that would result it in falling over and dying once it figures out how to modify itself does not imply that all possible intelligences would behave similarly. Somehow, humans are able to "want to be happy" while still refusing to take heroin or pain killers, and we have no reason yet to believe that AI is fundamentally unable to grasp this distinction between "stuff I care about" and "the different stuff I would care about if I changed my own design".