Earlier quoted context omitted.
> I see no reason why AI would have any difficulty defining its own reward functions The first problem is epistemological... If you think that creative decisions are made by complying with a "reward function," you are entirely missing something. Most values are fundamentally based on irrationality. I've literally spent an entire life doing things that everyone else told me was wrong and being interested in things tha…
I think perhaps you're just misunderstanding some of the terms you're attempting to use. Those things that everyone else told you were wrong and that no one else saw the value in... your reward function rewards pursuing those. And in that context your decisions were rational and functional.
What YOU have to realize is that you (like many others in the past) can only seem to understand the explanation for something in terms of only what is already understood. And that there is nothing "magical" or "special" about our current understanding (unless you believe there's nothing new to discover, which is preposterous hubris).