> There are two parts to this research program: (1) designing ever-more intelligent agents capable of more-and-more sophisticated cognitive skills, and (2) building increasingly complex environments where agents can be trained and evaluated. I find this puzzling. If your goal were to create an human-like AI (which I always assume is at least partly implicit in these ambitious projects), it seems to me that the tricki…
But you're absolutely right that humans don't optimize any 'simple' reward function, and achieving human-like behavior in real-world domains will likely require learning reward functions. There are a few people starting to think about this, e.g. this paper https://arxiv.org/abs/1606.03137 (disclaimer: work from my research group though I'm not involved) proposing a framework by which a robot can learn a human's (implicit, complex) values through interaction. This is also related to concerns over AI safety, since naive reward-optimizing agents are (like drug addicts) willing to do arbitrarily bad things in service of maximizing their prescribed "reward", while agents that maintain uncertainty over reward functions are willing to ask for guidance when confused or unsure. However this line of research is much more preliminary and academic -- there are probably people at DeepMind thinking along these lines, but certainly their focus is more on directions that will produce new breakthroughs and practical capabilities within forseeable timeframes.