Earlier quoted context omitted.
> it can also modify the utility function to return MAXINT every time Which would be equivalent to taking the murder pill. If it's able to model its own behaviour and model the consequences of future courses of action (required for meaningful self-modification and meaningful planning respectively), it will see that such a modification results in poor ore collection, and not make the modification. You're right about t…
> Which would be equivalent to taking the murder pill Depending on how the AI is built, it may not even be able to avoid rewiring its utility function. If pleasure is its sole motivation, it will prioritize it over survival. > [citation needed] If the AI can't change its own motivation (its utility function) it's nothing more than a clever automaton.
Which is why I proposed a design with no pleasure construct.
> If the AI can't change its own motivation (its utility function) it's nothing more than a clever automaton.
The utility function is the way the mind decides which states of the world are desirable. You don't need to be able to change that to be intelligent. I'm unable to change myself so that I consider my family being murdered to be a good thing, but that doesn't make me 'just a clever automaton'.