Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

AlphaGo Zero: Learning from scratch

deepmind.com

Post reply on HN
Print view View original

71–80 of 324 posts

Re: AlphaGo Zero: Learning from scratch

#71

Post by shmageggy »

Looks like the performance improvement comes from two key ingredients:

1) Using Residual networks instead of normal convolutional layers

2) Using a smarter policy training loss that uses the full information from a MCTS at each move. In the previous version, I believe they just ran the policy network to the end of the game and used a very weak {0, 1} reinforcement signal over all of the moves played. Here, it looks like they use each run of MCTS to provide a fully supervised signal over all moves it explores.

Re: AlphaGo Zero: Learning from scratch

#72

Post by shmageggy »

aeleos wrote: post #52

The fact that they only used self play with no outside input here is really interesting. I wonder if this system produced more new styles of play. While I am not that familiar with Go, I know in some of the other articles they talk about things like Chinese starts that are specific to certain cultures. I wonder if the fact that it had no outside reinforcement made it produce movements that we have already seen that a…

The key part from the paper:

> To assess the merits of self-play reinforcement learning, compared to learning from human data, we trained a second neural network (using the same architecture) to predict expert moves in the KGS Server data­ set; this achieved state-of-the-art prediction accuracy compared to pre­ vious work 12,30–33 (see Extended Data Tables 1 and 2 for current and previous results, respectively). Supervised learning achieved a better initial performance, and was better at predicting human professional moves (Fig. 3). Notably, although supervised learning achieved higher move prediction accuracy, the self-learned player performed much better overall, defeating the human-trained player within the first 24 h of training. This suggests that AlphaGo Zero may be learning a strategy that is qualitatively different to human play.

Re: AlphaGo Zero: Learning from scratch

#73

Post by postnihilism »

nicklovescode wrote: post #69

Anyone know why they trained on TPUs? My understanding is that the main benefit of a TPU is inference. Is this not true?

The first generation TPUs were focused on inference. Second generation TPUs are focused on both training and inference.

https://www.blog.google/topics/google-cloud/google-cloud-off...

Re: AlphaGo Zero: Learning from scratch

#74

Post by shmageggy »

greysphere wrote: post #45
davidkuhta wrote: post #3

> Previous versions of AlphaGo initially trained on thousands of human amateur and professional games to learn how to play Go. AlphaGo Zero skips this step and learns to play simply by playing games against itself, starting from completely random play. So technically this version has lost every game it's ever won. Jokes aside, it's pretty interesting to note that they were able to combine the "policy" and "value" net…

TD-gammon is a well known version of this technique (with 2 ply lookahead, vs a 1600 deep mcts) https://en.m.wikipedia.org/wiki/TD-Gammon Temporal difference learning was previously consider weak at 'tactical' games, ie ones with gamestates that require long chains of precise moves to improve position (like many checkmate scenarios in chess) . For anyone more familiar with this technique, is it clear how the mcts/che…

Yeah it's not clear to me why temporal difference learning all of a sudden works so well here? Is it the case that nobody had really tried it for learning a policy for Go with a strong NN architecture? In the Methods they mention TD learning for value functions but I don't see anything about policies.

edit: OK, they're calling it policy iteration as opposed to TD learning. I guess I don't get the difference.

Re: AlphaGo Zero: Learning from scratch

#75

Post by marcosscriven »

aeleos wrote: post #52

The fact that they only used self play with no outside input here is really interesting. I wonder if this system produced more new styles of play. While I am not that familiar with Go, I know in some of the other articles they talk about things like Chinese starts that are specific to certain cultures. I wonder if the fact that it had no outside reinforcement made it produce movements that we have already seen that a…

Possibly a dumb q, but is ‘self play’ in any way related to ‘adversarial’ learning? I don’t see it mentioned in the article, but it reminds me of the principle.

Re: AlphaGo Zero: Learning from scratch

#76

Post by davidkuhta »

greysphere wrote: post #45
davidkuhta wrote: post #3

> Previous versions of AlphaGo initially trained on thousands of human amateur and professional games to learn how to play Go. AlphaGo Zero skips this step and learns to play simply by playing games against itself, starting from completely random play. So technically this version has lost every game it's ever won. Jokes aside, it's pretty interesting to note that they were able to combine the "policy" and "value" net…

TD-gammon is a well known version of this technique (with 2 ply lookahead, vs a 1600 deep mcts) https://en.m.wikipedia.org/wiki/TD-Gammon Temporal difference learning was previously consider weak at 'tactical' games, ie ones with gamestates that require long chains of precise moves to improve position (like many checkmate scenarios in chess) . For anyone more familiar with this technique, is it clear how the mcts/che…

Wow, that was a really deep and enjoyable Wikipedia rabbit hole journey. I hadn't heard of Temporal Difference before (though I was familiar with Q-learning).

It was interesting to note that TD-Gammon improved with expert designed features. I wonder if this was simply related to the technology of the field as it stood over 20 years ago or some underlying categorization or complexity associated with the games themselves (backgammon being more favorable to human comprehension than Go in this case).

> Even though TD-Gammon discovered insightful features on its own, Tesauro wondered if its play could be improved by using hand-designed features like Neurogammon's. Indeed, the self-training TD-Gammon with expert-designed features soon surpassed all previous computer backgammon programs. It stopped improving after about 1,500,000 games (self-play) using 80 hidden units.

For others: Richard Sutton, one of the pioneers of TD makes his Reinforcement Learning: An Introduction textbook available for free on his website: http://incompleteideas.net/sutton/ (MIT Press also links to it)

Re: AlphaGo Zero: Learning from scratch

#77

Post by smdz »

Deep reinforcement learning is interesting and has plenty of potential. But highlighting AlphaGo as an example of reinforcement learning is like undermining the concepts of reinforcement learning.

Re: AlphaGo Zero: Learning from scratch

#78

Post by shmageggy »

davidkuhta wrote: post #76
greysphere wrote: post #45

Earlier quoted context omitted.

TD-gammon is a well known version of this technique (with 2 ply lookahead, vs a 1600 deep mcts) https://en.m.wikipedia.org/wiki/TD-Gammon Temporal difference learning was previously consider weak at 'tactical' games, ie ones with gamestates that require long chains of precise moves to improve position (like many checkmate scenarios in chess) . For anyone more familiar with this technique, is it clear how the mcts/che…

Wow, that was a really deep and enjoyable Wikipedia rabbit hole journey. I hadn't heard of Temporal Difference before (though I was familiar with Q-learning). It was interesting to note that TD-Gammon improved with expert designed features. I wonder if this was simply related to the technology of the field as it stood over 20 years ago or some underlying categorization or complexity associated with the games themselv…

PSA: The new edition of Sutton/Barto has a nice discussion of (the original) AlphaGo in the back.

http://incompleteideas.net/sutton/book/the-book-2nd.html

Re: AlphaGo Zero: Learning from scratch

#79

Post by pinouchon »

habitue wrote: post #30
Invictus0 wrote: post #6

Is this evidence of a broader leap forward in machine learning, or are these advancements domain-specific? In other words, could these innovations be applied to other fields and applications?

I think the fact that it's no longer using Monte Carlo tree search is a huge step forward in the generalizability of the technique. But go is still - a perfect information game - with a relatively small input size (vs. arbitrary computer vision) - cheap to simulate - discrete action space - deterministic This isn't to take away from the magnitude of the achievement, but the nature of the problem itself makes the resu…

Are you sure the new version is not using MCTS ? I saw it mentioned in the new paper

Re: AlphaGo Zero: Learning from scratch

#80

Post by skinner_ »

Jyaif wrote: post #4

"It uses one neural network rather than two." and "AlphaGo Zero only uses the black and white stones from the Go board as its input, whereas previous versions of AlphaGo included a small number of hand-engineered features." This is amazing! The technology they came up with must be super generic.

Also, unsupervised. Also, no rollouts. They got rid of a lot of complexity. At this point it looks like a reasonable challenge to write a superhuman Go AI in 500 lines of unobfuscated python.
Post reply on HN