Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Human-level control through deep reinforcement learning

nature.com

Post reply on HN
Print view View original

31–40 of 63 posts

Re: Human-level control through deep reinforcement learning

#31

Post by chriswarbo »

bmh100 wrote: post #7

> ...the authors used the same algorithm, network architecture, and hyperparameters on each game... This is huge. It shows that the algorithm was able to generalize across multiple problem sets within the same domain of "playing Atari 2600 games", and not simply a "lucky" choice of algorithm, network architecture, or hyperparameters that a random search for each game might choose. This is also not a violation of the…

The No Free Lunch Theorems don't apply in reality anyway, as explained at https://en.wikipedia.org/wiki/No_free_lunch_in_search_and_op...

Essentially: NFL says the performance (1 / number of trials are required to find a solution), when averaged over all fitness functions, will be the same for all optimisation algorithms. However, most fitness functions require more computational power than the Universe to evaluate; making even a single trial impossible to perform.

Since those problems are already impossible, it doesn't matter how many trials our optimisation algorithm would require when tackling them. We can use this unobservable bad performance to offset some observable good performance, to remain within the average dictated by NFL whilst still becoming objectively better.

In fact, one of this paper's authors (Shane Legg) has used this fact to define a general intelligence test which favours performance on simple functions http://arxiv.org/abs/0712.3329

Re: Human-level control through deep reinforcement learning

#32

Post by chriswarbo »

Someone wrote: post #29

For comparison: http://www.cs.cmu.edu/~tom7/mario/ . That is way more of a hack, but I am not sure this is that big a step forward. Space invaders and breakout aren't the hardest games and I haven't heard a hard argument that it is just a matter of scale to create a machine that, say, plays chess.

That's pretty cool (especially the ending of Tetris!), but the crucial difference is that it's trying to learn by observing what happens during a known-good run, then it sounds like some game-specific learning is performed. In contrast, the Arcade Learning Environment (the benchmark used in TFA) is completely autonomous; there is no human guidance or game-specific learning: anything it learns must work well across all of the games (ie. it's either generally useful, or it gets surrounded by a bunch of conditionals).

In particular, the author of your link says he's surprised that the algorithm can sometimes get further than twice the length of his pre-recorded training sequence.

Re: Human-level control through deep reinforcement learning

#33

Post by ratsimihah »

dwaltrip wrote: post #30
sjtrny wrote: post #16

Watch it play: http://www.nature.com/nature/journal/v518/n7540/extref/natur... http://www.nature.com/nature/journal/v518/n7540/extref/natur...

This is so cool. I'd love to work on this stuff... Anyone know how hard it would be for someone who is fairly good at programming (works as a full stack developer and feels quite comfortable learning new things) and has strong math skills (undergrad degree) to break into this field? Is going back to school for a masters/phd the best way?

How about you read the article and get a few books on the relevant topics? It would probably be much cheaper than going back to school.

Re: Human-level control through deep reinforcement learning

#34

Post by devindotcom »

nl wrote: post #2

Is this a different paper to the original DeepMind video game paper? http://arxiv.org/abs/1312.5602

Yes, they addressed this on the conference call I was on. That was a sort of 'initial results' they wanted to get out there, this is the full research. Same concept, explored more deeply.

Re: Human-level control through deep reinforcement learning

#35

Post by TheEzEzz »

Someone wrote: post #29

For comparison: http://www.cs.cmu.edu/~tom7/mario/ . That is way more of a hack, but I am not sure this is that big a step forward. Space invaders and breakout aren't the hardest games and I haven't heard a hard argument that it is just a matter of scale to create a machine that, say, plays chess.

The biggest differences are this:

1. The Mario algo has direct access to the game state, and will only work for games where it has that game state access. The DeepMind algo plays directly from the screen pixels. That means DeepMind has to first learn to interpret time varying (!) visual information correctly, then deduce rules and good play strategy on top of that leaky abstraction. That's hard. It also means the algorithm can be applied to any game with a screen output, not just to an Atari.

2. The Mario algo is doing a direct search through move space. It can back up and explore a different branch of the tree and play differently to see a different outcome. When the DeepMind algo plays Atari, it can't undo a move that it just did. It has to make good choices, using intelligence, just like a human player would.

The impressive thing here is not that it plays Atari games. You're right, we have had AIs that can do this for a long time, even better than this. The impressive thing is that it's a single AI algorithm that works for many games, and that is learning directly from the screen. We have not had anything like this before.

Re: Human-level control through deep reinforcement learning

#36

Post by chriswarbo »

j_m_b wrote: post #15
brentjanderson wrote: post #12

Earlier quoted context omitted.

Here is a very rough translation from my POV: > The theory of reinforcement learning provides a normative account, deeply rooted in psychological and neuroscientific perspectives on animal behaviour, of how agents may optimize their control of an environment. Reinforcement (rewards/punishments) is a highly effective way to train autonomous individuals to succeed in arbitrary environments. > To use reinforcement learn…

|> This work bridges the divide between high-dimensional sensory inputs and actions, resulting in the first artificial agent that is capable of learning to excel at a diverse array of challenging tasks. |We've created a universal learning algorithm that can take a multitude of inputs and consistently respond correctly without having to re-define the model for each game or problem. Further along in the article, it tur…

> Further along in the article, it turns out it is actually a subset of games which don't involve long term planning strategies.

Unfortunately I can't access the paper, and the ReadCube link seems to require Flash, but I'm assuming a "Q-network" is based on Q-learning. Q-learning is essentially a lookup table of states to actions, where we guess what state we're in from our observations and look up which action to perform that will get us the most reward.

In that sense, it's clear that a Q-learning approach would struggle with long-term planning, since its memory only goes as far as 1 action. Of course there are ways to extend Q-learning, but these tend to destroy it's best feature: implementation efficiency.

One nice alternative I've seen in recent years is Gradient Temporal Difference, which allows linear functions to be learned rather than just single actions, and retains lots of the performance properties of Q-learning (O(n) in the number of functions learned, off-policy learning, etc.).

Re: Human-level control through deep reinforcement learning

#37

Post by ZeroFries »

bsdetector wrote: post #27
bmh100 wrote: post #7

> ...the authors used the same algorithm, network architecture, and hyperparameters on each game... This is huge. It shows that the algorithm was able to generalize across multiple problem sets within the same domain of "playing Atari 2600 games", and not simply a "lucky" choice of algorithm, network architecture, or hyperparameters that a random search for each game might choose. This is also not a violation of the…

> the algorithm was able to generalize across multiple problem sets Did it really? I think they reset it and retrained it for each game. I'd like to know how much more is needed to make one instance of the AI that can successfully play any of the games. To play all 49 games that it could learn, does it need to be an extra level deep? Or 49 times larger? Or 2^49 times more?

I think you could establish an upperbound at 50 times larger, because each set of "neurons" could be completely independent for each game, with 49 neurons to determine which set to use.

If they have similar features (which they undoubtedly would), the size could drop a lot.

Of course, I have no training in neural nets, and so my conclusions are reached from general understanding/reasoning.

Re: Human-level control through deep reinforcement learning

#38

Post by dwaltrip »

ratsimihah wrote: post #33
dwaltrip wrote: post #30

Earlier quoted context omitted.

This is so cool. I'd love to work on this stuff... Anyone know how hard it would be for someone who is fairly good at programming (works as a full stack developer and feels quite comfortable learning new things) and has strong math skills (undergrad degree) to break into this field? Is going back to school for a masters/phd the best way?

How about you read the article and get a few books on the relevant topics? It would probably be much cheaper than going back to school.

Very good point. I taught myself web dev (now working at a pretty awesome startup) so I'm definitely familiar with that route.

I have a few cool AI ideas I'm hoping to start spending more time in the coming months, and I have heard of some great online courses to check out. I was just curious as to how important institutional credentials are for this kind of thing, seeing as it much more academic than building CRUD web apps.

Re: Human-level control through deep reinforcement learning

#39

Post by leereeves »

jmnhr wrote: post #22
frandroid wrote: post #21

Earlier quoted context omitted.

Let it load...

They are using tricks. When I tried the first time the pdf was blurred and the page automatically opened the payment menu. On the second try it showed the entire pdf, disallowing download.

It loaded completely for me the first time, but doesn't allow download.

A step toward open access, but they're still trying to claim copyright and control the work of others.

Re: Human-level control through deep reinforcement learning

#40

Post by DavidSJ »

bsdetector wrote: post #27
bmh100 wrote: post #7

> ...the authors used the same algorithm, network architecture, and hyperparameters on each game... This is huge. It shows that the algorithm was able to generalize across multiple problem sets within the same domain of "playing Atari 2600 games", and not simply a "lucky" choice of algorithm, network architecture, or hyperparameters that a random search for each game might choose. This is also not a violation of the…

> the algorithm was able to generalize across multiple problem sets Did it really? I think they reset it and retrained it for each game. I'd like to know how much more is needed to make one instance of the AI that can successfully play any of the games. To play all 49 games that it could learn, does it need to be an extra level deep? Or 49 times larger? Or 2^49 times more?

What they mean by "the algorithm" is the learning algorithm, not the actual learned behavior. Historically in machine learning a great deal of hand tuning of learning algorithms is required for them to even be effective on a single problem, so it's impressive that a single algorithm was able to learn to play many different games effectively.

Having a single instance of an algorithm learn to perform well on many different tasks is really a separate problem than what they're dealing with here.

Post reply on HN