Searle's 'experiment' is silly. Searle is trying to make readers empathize with the 'person in the box'. Let's concede, for the moment, that the person in the box doesn't understand Chinese in any meaningful way. That says nothing about whether or not the SYSTEM on a WHOLE understands Chinese. Given the constraints of Searle's example (that every input produces a 'correct' output, but does so via this labour-intensiv…
Since human motivations and preferences are caused by evolutionary forces, I don't see how an AI would acquire such things, aside from what we build into it. And if we wanted to build motivation into it, it shouldn't be difficult, once we have an artificial mind capable of understanding the necessary concepts.
2) A lot of AI research is premised upon modeling selection between choices. It seems entirely possible that the questions I have would be incredibly boring compared to whatever other thoughts are there to be had. That is: what's useful to us might not be coequal with AI, and that limited, domain-specific pseudo-intelligence is really what's necessary. But any time there are external constraints on exploration space (for instance, requiring a response within a certain time), we don't know what we're missing out on.