AlphaGo Zero: Learning from scratch • Hacker News Board

Re: AlphaGo Zero: Learning from scratch

#41

Post by EvgeniyZh » Wed, Oct 18, 2017, 6:26 PM UTC

Florin_Andrei wrote: post #25

galkk wrote: post #21

Why don't use the same approach for chess? It's very interesting to see if it is able to handle much more advanced and tuned engines that exist for chess, game with considerable much more complicated rules?

I think chess is less compelling because, in a sense, it is a "solved problem" - superhuman AI chess players already exist. And chess, while it does have more complex base rules, has a much lower combinatorial complexity than Go.

Well, I'd love to see NN solution beating top chess engines. It might also introduce novelty to the game, just as regular engines did

Re: AlphaGo Zero: Learning from scratch

#42

Post by alexbeloi » Wed, Oct 18, 2017, 6:28 PM UTC

Very impressive, the original implementation relied a lot on feature engineering.

I'm surprised they're able to prevent a self-play equilibrium with such a simple loss function.

It's sort of like they are using auxiliary outputs but instead of using them to fit features, they are fitting to multiple ways of arriving at 'best play', through predicting value (SL) and predicting probability for best outcome (RL). In principle, they're doing the same thing but in practice it seems like they are making up for each others shortcomings (e.g. self-play equilibrium with RL).

Re: AlphaGo Zero: Learning from scratch

#43

Post by apetresc » Wed, Oct 18, 2017, 6:28 PM UTC

davidkuhta wrote: post #3

> Previous versions of AlphaGo initially trained on thousands of human amateur and professional games to learn how to play Go. AlphaGo Zero skips this step and learns to play simply by playing games against itself, starting from completely random play. So technically this version has lost every game it's ever won. Jokes aside, it's pretty interesting to note that they were able to combine the "policy" and "value" net…

> So technically this version has lost every game it's ever won.

No, they've also played it against AlphaGo Lee and AlphaGo Master. The SGFs are available at: https://www.nature.com/nature/journal/v550/n7676/extref/natu...

Re: AlphaGo Zero: Learning from scratch

#44

Post by Radim » Wed, Oct 18, 2017, 6:28 PM UTC

habitue wrote: post #30

Invictus0 wrote: post #6

Is this evidence of a broader leap forward in machine learning, or are these advancements domain-specific? In other words, could these innovations be applied to other fields and applications?

I think the fact that it's no longer using Monte Carlo tree search is a huge step forward in the generalizability of the technique. But go is still - a perfect information game - with a relatively small input size (vs. arbitrary computer vision) - cheap to simulate - discrete action space - deterministic This isn't to take away from the magnitude of the achievement, but the nature of the problem itself makes the resu…

Except it is still using MCTS (heavily).

Re: AlphaGo Zero: Learning from scratch

#45

Post by greysphere » Wed, Oct 18, 2017, 6:30 PM UTC

davidkuhta wrote: post #3

> Previous versions of AlphaGo initially trained on thousands of human amateur and professional games to learn how to play Go. AlphaGo Zero skips this step and learns to play simply by playing games against itself, starting from completely random play. So technically this version has lost every game it's ever won. Jokes aside, it's pretty interesting to note that they were able to combine the "policy" and "value" net…

TD-gammon is a well known version of this technique (with 2 ply lookahead, vs a 1600 deep mcts) https://en.m.wikipedia.org/wiki/TD-Gammon

Temporal difference learning was previously consider weak at 'tactical' games, ie ones with gamestates that require long chains of precise moves to improve position (like many checkmate scenarios in chess) .

For anyone more familiar with this technique, is it clear how the mcts/checkpoint system overcomes this? How sensative is the system to the tuning params for those parts of the alg. Like is Go a particularly good candidate because of the ~400 play positions resulting in a (relatively) small tree seach requirement? (I kinda cant believe im saying that go has 'a small search tree'!)

We us td learning for the ai in our game Race for the Galaxy, so it's neat to hear about possible avenues for improvement!

Re: AlphaGo Zero: Learning from scratch

#46

Post by Scaevolus » Wed, Oct 18, 2017, 6:31 PM UTC

gordon_freeman wrote: post #33

Jyaif wrote: post #4

"It uses one neural network rather than two." and "AlphaGo Zero only uses the black and white stones from the Go board as its input, whereas previous versions of AlphaGo included a small number of hand-engineered features." This is amazing! The technology they came up with must be super generic.

I will be interested to see what kind of algorithms they have used to allow AlphaGo to learn from its own moves. Are these pretty generics algos or are these very customized and specific ones that only apply to AlphaGo and the game of Go?

They have a new reinforcement learning algorithm that should be generically applicable to anything where a long sequence of moves results in a specifically gradable outcome.

> The neural network in AlphaGo Zero is trained from games of selfplay by a novel reinforcement learning algorithm. In each position s, an MCTS search is executed, guided by the neural network fθ. The MCTS search outputs probabilities π of playing each move. These search probabilities usually select much stronger moves than the raw move probabilities p of the neural network fθ(s); MCTS may therefore be viewed as a powerful policy improvement operator. Self-play with search—using the improved MCTS-based policy to select each move, then using the game winner z as a sample of the value—may be viewed as a powerful policy evaluation operator. The main idea of our reinforcement learning algorithm is to use these search operators repeatedly in a policy iteration procedure: the neural network’s parameters are updated to make the move probabilities and value (p, v)= fθ(s) more closely match the improved search probabilities and selfplay winner (π, z); these new parameters are used in the next iteration of self-play to make the search even stronger.

Re: AlphaGo Zero: Learning from scratch

#47

Post by orbifold » Wed, Oct 18, 2017, 6:31 PM UTC

galkk wrote: post #21

Why don't use the same approach for chess? It's very interesting to see if it is able to handle much more advanced and tuned engines that exist for chess, game with considerable much more complicated rules?

Denis Hassabis was asked this in a Q&A after a talk he gave and according to him someone did this (bootstrap a chess engine from self play) successfully, while still being a student and was hired by them subsequently.

Re: AlphaGo Zero: Learning from scratch

#48

Post by amelius » Wed, Oct 18, 2017, 6:31 PM UTC

Jyaif wrote: post #4

"It uses one neural network rather than two." and "AlphaGo Zero only uses the black and white stones from the Go board as its input, whereas previous versions of AlphaGo included a small number of hand-engineered features." This is amazing! The technology they came up with must be super generic.

But I suppose they still do the searching/pruning with a separate piece of code (not a neural network).

Re: AlphaGo Zero: Learning from scratch

#49

Post by Tarq0n » Wed, Oct 18, 2017, 6:31 PM UTC

cjbprime wrote: post #29

panic wrote: post #26

Earlier quoted context omitted.

MCTS means "Monte-Carlo Tree Search". It's the core of the algorithm. The big difference is that it doesn't use rollouts, or random play: it chooses where to expand the tree based only on the neural network.

No, 'habitue is correct. This new blog post says that the new software no longer does game readouts and just uses the neural net.

That's not what Monte Carlo Tree search is. The new version is still one neural network + MCTS. There's no way to store enough information to judge the efficiency of every possible move in a neural network, therefore a second algorithm to simulate outcomes is necessary.

Re: AlphaGo Zero: Learning from scratch

#50

Post by hacker_9 » Wed, Oct 18, 2017, 6:36 PM UTC

amelius wrote: post #48

Jyaif wrote: post #4

"It uses one neural network rather than two." and "AlphaGo Zero only uses the black and white stones from the Go board as its input, whereas previous versions of AlphaGo included a small number of hand-engineered features." This is amazing! The technology they came up with must be super generic.

But I suppose they still do the searching/pruning with a separate piece of code (not a neural network).

From the paper: "it uses a single neural network, rather than separate policy and value networks ... it uses a simpler tree search that relies upon this single neural network to evaluate positions and sample moves, without performing any MonteCarlo rollouts. To achieve these results, we introduce a new reinforcement learning algorithm that incorporates lookahead search inside the training loop, resulting in rapid improvement and precise and stable learning."