A robot has to decide is the show must go on or not.
Is Intelligence Self-Limiting?
71–80 of 81 posts
Re: Is Intelligence Self-Limiting?
#72Earlier quoted context omitted.
It only takes a tiny percentage to turn down said machine. If 10 billion people plug-in, but 1 million abstain and continue to evolve past the 10 billion, the majority stagnates. Those 1 million will eventually gain a competitive advantage over the 10 billion, evolve past them, and likely extinct them. -------------------- Prediction: Humans won't be around in 1,000 years if technology progresses at current rates. Su…
Individuals (10 billion/1 million) do not evolve - populations evolve. Evolution takes a long time, and even the spectacularly obvious gross phenotypic changes you see between various breeds of dog, occuring over the last 10,000 years or so, has not produced any actual speciation yet. And really, unless the amount of interbreeding between two "separate" populations is infinitesimally small (or actually zero, because…
If this were slashdot, I could say that it's already happened ;)
Re: Is Intelligence Self-Limiting?
#73What's the purpose of this 'pleasure' construct in the AI's mind? If it's able to calculate the value of a utility function in order to set the 'pleasure' variable, and it bases its actions on the value of this 'pleasure' variable, why not just cut out the middle man and have it base its actions directly on the result of the utility function? The variable functionally buys you nothing, but introduces this problem tha…
I think it would work better if when self-modifying, it ran simulations on the result with its current utility function, but there would be a problem if it did end up modifying the utility function or modified away the simulation requirement. It's hard to balance the ability to create new, more useful utility functions with prevention of creating a utility function at odds with what the original entity valued.
> It's hard to balance the ability to create new, more useful utility functions with prevention of creating a utility function at odds with what the original entity valued.
I think that for such an AI, the concept of a "more useful utility function" would be a nonsense. The AI's definition of 'useful' is the utility function. No other utility function can ever rate higher against the current utility function than the current utility function does.
Re: Is Intelligence Self-Limiting?
#74What's the purpose of this 'pleasure' construct in the AI's mind? If it's able to calculate the value of a utility function in order to set the 'pleasure' variable, and it bases its actions on the value of this 'pleasure' variable, why not just cut out the middle man and have it base its actions directly on the result of the utility function? The variable functionally buys you nothing, but introduces this problem tha…
> why not just cut out the middle man and have it base its actions directly on the result of the utility function? If the autonomous robot can modify its own programming, it can also modify the utility function to return MAXINT every time. In fact, being able to modify the utility function is a pre-requisite to be called intelligent. One way to counter this is to create long and short-term utility functions so that t…
> it can also modify the utility function to return MAXINT every time
Which would be equivalent to taking the murder pill. If it's able to model its own behaviour and model the consequences of future courses of action (required for meaningful self-modification and meaningful planning respectively), it will see that such a modification results in poor ore collection, and not make the modification.
You're right about the time-envelope of the utility function being an issue. The AI needs to plan far enough ahead at all times to see all relevant consequences of its actions. I don't think that requires two separate utility functions though, a single long-term one should do the job.
Edit: Also, "being able to modify the utility function is a pre-requisite to be called intelligent."? [citation needed]
Re: Is Intelligence Self-Limiting?
#75This is called "wireheading", and while it's an interesting possibility, unfortunately what we know indicates that AIs won't do it. See Stephen Omohundro, "The Basic AI Drives" ( http://selfawaresystems.files.wordpress.com/2008/01/ai_drive... ). The basic argument is that a sufficiently intelligent mind will make a distinction between the world being a certain way, and it perceiving that the world is a certain way, a…
Louis Wu's tasp in Ringworld also comes to mind.
Re: Is Intelligence Self-Limiting?
#76Iirc, there was an article a while back (or maybe a link from a comment) about how video games were the real reason behind the fermi paradox. The argument was that a civilization's entertainment would constantly be improving to the point where we could jack into a self-controlled matrix-like world where we had complete control and, in essence, are deities in our self-fabricated worlds. If we built robots to ensure ou…
It only takes a tiny percentage to turn down said machine. If 10 billion people plug-in, but 1 million abstain and continue to evolve past the 10 billion, the majority stagnates. Those 1 million will eventually gain a competitive advantage over the 10 billion, evolve past them, and likely extinct them. -------------------- Prediction: Humans won't be around in 1,000 years if technology progresses at current rates. Su…
"It only takes a tiny percentage to turn down said machine. If 10 billion people plug-in, but 1 million abstain and continue to evolve past the 10 billion, the majority stagnates. Those 1 million will eventually gain a competitive advantage over the 10 billion, evolve past them, and likely extinct them."
If you look at the distribution of wealth, there's a pretty good argument this already happened.
Re: Is Intelligence Self-Limiting?
#77Earlier quoted context omitted.
It's not a straw man, it's a thought experiment. Obviously you can't actually do it. I'm sure you'd have trouble filling a room with rules about how to handle chinese symbols, or teaching a colourblind scientist everything it's possible to know about colour. It's hypothetical, it's a thought experiment. Your first criticism is just invalid. The pill rewrites the workings of your brain to enjoy murder, the experience…
Its not invalid. As a normal human who hasn't used that hypothetical drug, I find it repugnant. That was the point you were making: you will choose not to take that pill. No matter how enjoyable and good murder may be for you if you take the pill, your own self-knowledge and current utility function prohibit taking it. The reason it is a straw man is that it is an extreme example that is not realistic. Here is a real…
> what if you could plug a cable into your brain and experience your wildest dreams, as vividly as real life?
Tough one. Even if everyone is offered the same choice, I'm not sure I'd take it, because I (think I) value actual interaction with my peers (I trust the brain stimulator could provide realistic fake interaction).
The point is, if I really valued internal stimulation only, I would plug into the machine in a heartbeat. But I do care about the outside world, so I'd probably wouldn't do that. That's why I don't think it is impossible to build an AI that actually makes sure it optimizes the outside world, instead of mere internal reward signals.
Re: Is Intelligence Self-Limiting?
#78The problem is that the instincts that humans possess today, (ie, maximizing short-term comfort, possibly at the expense of long-term survival) is what got us this far. That cannot suddenly be decoupled and discarded as useless simply because we now live in a remarkably stable environment that enables us to focus on things other than immediate survival.
I suspect that if there were some horrible catastrophe and the human race were suddenly thrown back into an archaic society without any of the technological advancements that we have at our disposal today, it would be those MOST focused on short-term gain who would be most likely to perpetuate their own, and consequently, human existence.
Re: Is Intelligence Self-Limiting?
#79Earlier quoted context omitted.
> why not just cut out the middle man and have it base its actions directly on the result of the utility function? If the autonomous robot can modify its own programming, it can also modify the utility function to return MAXINT every time. In fact, being able to modify the utility function is a pre-requisite to be called intelligent. One way to counter this is to create long and short-term utility functions so that t…
> it can also modify the utility function to return MAXINT every time Which would be equivalent to taking the murder pill. If it's able to model its own behaviour and model the consequences of future courses of action (required for meaningful self-modification and meaningful planning respectively), it will see that such a modification results in poor ore collection, and not make the modification. You're right about t…
> Which would be equivalent to taking the murder pill
Depending on how the AI is built, it may not even be able to avoid rewiring its utility function. If pleasure is its sole motivation, it will prioritize it over survival.
> [citation needed]
If the AI can't change its own motivation (its utility function) it's nothing more than a clever automaton.
Re: Is Intelligence Self-Limiting?
#80So, drugs are just as effective in AIs as they are in human intelligences.
The author calls drugs "primitive pre-FOO hacks", but since they already have such a powerful effect on human intelligence, what's the point of differentiating between pre- and post-FOO hacks? One will just lead to self-destruction much quicker than the other.