I used to say that a key component of AI that was missing was the ability to get through the next few seconds of life without falling down or bumping into anything. I went through Stanford CS when the top-down logicians were in charge of AI. That approach was totally incapable of dealing with the real world. Now we're seeing the systems needed to deal with the real world in the short term starting to work.
Once you can deal with the next few seconds, a strategy module can be added to give goals to the low level system. This is very clear in the video game context. As the game playing programs advance beyond the 2D full-screen games, they'll need a low-level system to handle the next moves ("don't fall off platform", "jump to next platform", "shoot at target" are primitives for the 2D sidescroller era) and some level of planner to handle tactical matters and strategy.
It's possible to explicitly build hierarchical systems like that now, using classical planning techniques to modify the goals of a machine learning system. It's not yet possible to get a hierarchical system to emerge from machine learning. Medium term planning as an emergent behavior is a near term big challenge for AI.
Beyond such a two-level system, we're going to need intercommunicating components that do different parts of the problem. The components may be evolved, while the architecture may be designed. When AI systems can design such architectures, they're probably ready to take over.