Is Intelligence Self-Limiting?

by donnawarellp

  I have often thought that life is self limiting. For example, yeast is a
  living thing, it consumes sugars and it's waste is ehtanol and eventually
  they all kill themselves off by drowning in their own waste (which, I
  guess later over a meal I will be enjoying a glass of yeast pee, but I
  digress). From a molecular perspective, human DNA is not that different
  from yeast, and at a macro view our behaviour is not that different
  either it seems. Perhaps all sentient life is destine to the same fate,
  perhaps it is a law of nature. I recall reading about an argument that
  Stephen Hawking and Kip Thorne have regarding time travel. Stephen
  Hawking argues against time travel because we have never met any time
  travellers from our future. Maybe time travel is theoretically possible,
  but like yeast, we do not as a species survive long enough to develop
  the necessary technology. ugh, sorry to bring everyone down, might as
  well have a glass of yeast pee.

Using the expression "what you fundamentally value" is not the right way to put it. In real life, what you fundamentally value changes on a minute-by-minute basis. If you were able to ask a rat what he valued, he would probably say food and water, aka survival. However, as soon as they put this rat into a Skinner box, it literally pleases itself until it dies of exhaustion, even though food and water were available to it.

I agree with you that the best way to run an AI/VI like this would be to skip the idea of "pleasure" entirely and simply make productivity its' fundamental value. You would also set it up so that while it could modify its' own source code, it could not change what it valued. It seems like it would be relatively easy to set it up so that it would "hide" parts of its' code from itself and make it off-limits from modification.

Why would it want or not want anything, if it doesn't have a pleasure construct (which might also be called a motivation construct, since an AI might not be capable of the same subjective experience of pleasure that we are)?

I think it's a question of program design whether there's a utility function which decides whether to trigger the pleasure construct, or whether certain sensory input modules directly trigger the pleasure construct. To limit hacking potential, routing everything through a tamper-proof utility function might be better, except that it would also limit the AI's adaptability (short of recreating its own hardware to remove the tamper-proof module... which it might never do depending on the details of its motivation construct).

The reason it is a straw man is that it is an extreme example that is not realistic. Here is a realistic example: what if you could plug a cable into your brain and experience your wildest dreams, as vividly as real life?

But thats starting to get off your original point. Heres a funny story about what happens when you reward specific productivity measures: http://highered.blogspot.com/2009/01/well-intentioned-commis...

Personally I think that we have evolved pleasure centers specifically to avoid the pitfalls of hardwiring us to do things. If we were hardwired to procreate, it would be too easy to hack, but since we enjoy the process, it keeps us coming back.

But there's no particular reason to believe that the upper limit of intelligence is pretty much right where we are, either.

In fact, I'd argue that based on what we know about other evolved systems, our priors for "the biologically evolved system that does X has plenty of room for improvement" should be much, much, much higher than "the biologically evolved system for X is almost optimal for achieving X".

Like, a thousand to one or more - there are very few tasks that evolution has found optimal solutions to (which is not surprising, its "goal" is gene survival, everything else is an accident), and most of those are extremely low level physical things, chemical processes and the like. I'd need to hear some really strong argument that suggested that there's something special about human intelligence that should make us think it's near the limit, otherwise the odds ratio is just too hard to overcome...