What Google DeepMind Means for A.I.

Do you know how much 'state' DeepMind is tracking? Is it just choosing the best action given perceptions at the moment, or does it have some level of memory to work from? I wonder how DeepMind would do at path-finding in a maze for example - would it get stuck in an oscillating state?

The cynic in me would guess it's because he's not a big fan of neural nets, and prefers a more symbolic/statistical machine learning approach.

Personally, I don't think this is a dead end. Reinforcement learning with an effective method of representation learning is pretty much all you need for a general AI. Here, deep neural nets are able to learn how to represent the massive state space for vision quite well, but there's a bit of a mismatch in games where actions have to be taken in a specific order (e.g., where extended pathfinding is required) because for reinforcement learning to work well, your features need to capture that sort of temporally extended state information.

That's still a hard problem, but not insurmountable, and there's already strides in that direction. From the RL side, there's things like option models for taking series of actions, and from the deep learning side there's things like long short term memory and DeepMind's own neural Turing machines.

Knowing the group at DeepMind, they'll be able to crack it, and I think Mason is being entirely too pessimistic about the timeline in any event. Fifty years to control a drone?

> (there's no reasoning going on in the DeepMind work, just statistical correlation)

I've seen 100 people make this statement and mean 100 different things, so I just wanted to clarify:

How are you defining "reasoning" here as distinct from statistical correlation?

I really like this line of work and I expect will grow quite substantially over the next few years. Of course, Reinforcement Learning has been around for a long time. Similarly, Q Learning (the core model in this paper) has been around a very long time. What is new is that normally you see these models applied to toy MDP problems with simple dynamics, and linear Q function approximations for fear of non-convergence etc. What's novel about this work is that they fully embrace a complex non-linear Q function (ConvNet) looking at the raw pixels, and get it to actually work in (relatively speaking) complex environments (games). This requires several important tricks, as is discussed at length in their Nature paper (e.g. experience replay, updating the Q function only once in a while, etc.).

I implemented the DQN algorithm (used in this work) in Javascript a while ago as well (http://cs.stanford.edu/people/karpathy/convnetjs/demo/rldemo...) if people are interested in poking around, but my version does not implement all the bells and whistles.

The results in this work are impressive, but also too easy to antropomorphise. If you know what's going on under the hood you can start to easily list off why this is unlike anything humans/animals do. Some of the limitations include:

- Most curcially, the exploration used is random. You button mash random things and hope to receive a reward at some point or you're completely lost. If anything at any point requires a precise sequence of actions to get a reward, exponentially more training time is necessary.

- Experience replay that performs the model updates is performed uniformly at random, instead of some kind of importance sampling. This one is easier to fix.

- A discrete set of actions is assumed. Any real-valued output (e.g. torque on a join) is a non-obvious problem in the current model.

- There is no transfer learning between games. The algorithm always starts from scratch. This is very much unlike what humans do in their own problem solving.

- The agent's policy is reactive. It's as if you always forgot what you did 1 second ago. You keep repeatedly "waking up" to the world and get 1 second to decide what to do.

- Q Learning is model-free, meaning that the agent builds no internal model of the world/reward dynamics. Unlike us, it doesn't know what will happen to the world if it perfoms some action. This also means that it does not have any capacity to plan anything.

Of these, the biggest and most insurmountable problem is the first one: Random exploration of actions. As humans we have complex intuitions and an internal model of the dynamics of the world. This allows us to plan out actions that are very likely to yield a reward, without flailing our arms around greedily, hoping to get rewards at random at some point.

Games like Starcraft will significantly challenge an algorithm like this. You could expect that the model would develop super-human micro, but have difficulties with the overall strategy. For example, performing an air drop to enemy base would be impossible with the current model: You'd have to plan it out over many actions: "load the marines into the ship, fly the ship in stealth around the map, drop it at the precise location of enemy base".

Hence, DQN is best at games that provide immediate rewards, and where you can afford to "live in the moment" without much planning. Shooting things in space invaders is a good example. Despite all these shortcoming, these are exciting results!

Evolution didn't know, too. But it happened. And we easily find criteria saying that thing doesn't behave like a toddler. It would be a huge step forward to see a list of positive criteria.