Human-level control through deep reinforcement learning • Hacker News Board

Re: Human-level control through deep reinforcement learning

#61

Post by craftit » Thu, Feb 26, 2015, 4:44 PM UTC

phkahler wrote: post #41

eveningcoffee wrote: post #24

I am wondering that what kind of real life problems could be modelled this way.

>> I am wondering that what kind of real life problems could be modelled this way. Driving a car? Making a taco? Working the checkout counter?

Keeping someone entertained in a game, the longer they play the more reward you give it. I was also thinking of hooking up a face recognition system, so it can be rewarded when the user smiles.

Re: Human-level control through deep reinforcement learning

#62

Post by craftit » Thu, Feb 26, 2015, 4:51 PM UTC

eveningcoffee wrote: post #49

phkahler wrote: post #41

Earlier quoted context omitted.

>> I am wondering that what kind of real life problems could be modelled this way. Driving a car? Making a taco? Working the checkout counter?

So could you define an cost function for these activities?

I suspect setting up motivations for the AI is going to be a big research issue before too long. If you can write a simulator for the task you want to it solve, you should be able to train it. Often writing the simulator is much easier than solving a task itself. For example, the atlas robot, then can simulate it but struggle to control it.

Re: Human-level control through deep reinforcement learning

#63

Post by DavidSJ » Mon, Mar 02, 2015, 1:32 PM UTC

chriswarbo wrote: post #36

j_m_b wrote: post #15

Earlier quoted context omitted.

|> This work bridges the divide between high-dimensional sensory inputs and actions, resulting in the first artificial agent that is capable of learning to excel at a diverse array of challenging tasks. |We've created a universal learning algorithm that can take a multitude of inputs and consistently respond correctly without having to re-define the model for each game or problem. Further along in the article, it tur…

> Further along in the article, it turns out it is actually a subset of games which don't involve long term planning strategies. Unfortunately I can't access the paper, and the ReadCube link seems to require Flash, but I'm assuming a "Q-network" is based on Q-learning. Q-learning is essentially a lookup table of states to actions, where we guess what state we're in from our observations and look up which action to pe…

Q-learning is not a lookup table from states to actions. It's a learned mapping (using any method you want, such as neural networks) from agent history (i.e. the agent is allowed to consider the past) & action pairs, to estimates of future reward, learned through temporal differences between the prediction at time step t, and the sum of new prediction plus immediate reward at time step t+1.

In this way, information can flow back in time as the agent learns that some observation is predictive of reward, and then learns some observation is predictive of that observation, and so on until it connects the reward with some action K time steps ago. So long-term planning is definitely possible.

This does require that some relevant information be available at each intermediate time step to connect the actions with the ultimate reward. The nice thing about these Atari games is you can usually judge value just by what's immediately on the screen, and in this paper they only use the last three frames' worth of state. In a game requiring a memory for past sensory information (e.g. where information appears on the screen then disappears), this might not do so well, but that's more a matter of working memory than long-term planning, and a different Q-learning system could contain a working memory (e.g. if it was RNN-based).