Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How AlphaZero Mastered Its Games

newyorker.com

Post reply on HN
Print view View original

11–20 of 80 posts

Re: How AlphaZero Mastered Its Games

#11

Post by stabbles »

elcomet wrote: post #5
glinscott wrote: post #3

James put together a really nice summary of the ideas and the projects! It was almost a year ago that lc0 was launched, since then the community (led by Alexander Lyashuk, author of the current engine) has taken it to a totally different level. Follow along at http://lczero.org ! Gcp has also done an amazing job with Leela Zero, with a very active community on the Go side. http://zero.sjeng.org Of course, DeepMind re…

I thought alphazero was based on minimax, and used neural networks to evaluate moves, isn't this the case ?

Very roughly you can think of AlphaZero as a best-first tree search, where 'best' is some statistical estimate.

Re: How AlphaZero Mastered Its Games

#12

Post by stabbles »

What's very interesting is that the Komodo developers have implemented a Monte Carlo Tree Search version of their engine without neural nets for evaluation / move selection. This brand new engine can actually compete at the top level (still much worse than Stockfish and slightly worse than Lc0) [1] [2]

The exact implementation details are probably kept secret, but the idea is to do a few steps of minimax / alpha-beta rather than completely random play in the playout phase of MCTS.

This makes me think that the contribution of AlphaZero is not necessarily neural nets, but rather MCTS as a succesful method to search the game tree efficiently.

[1] http://tcec.chessdom.com/ [2] http://www.chessdom.com/komodo-mcts-monte-carlo-tree-search-...

Re: How AlphaZero Mastered Its Games

#13

Post by ggggtez »

stabbles wrote: post #12

What's very interesting is that the Komodo developers have implemented a Monte Carlo Tree Search version of their engine without neural nets for evaluation / move selection. This brand new engine can actually compete at the top level (still much worse than Stockfish and slightly worse than Lc0) [1] [2] The exact implementation details are probably kept secret, but the idea is to do a few steps of minimax / alpha-beta…

You missed the point then. Alpha beta pruning requires knowledge of the game rules. Neural network pruning doesn't. The advantage is that it's a general purpose technique.

Re: How AlphaZero Mastered Its Games

#14

Post by alan_wade »

God what a well written article! I don't have much to say on the subject, but this was pure joy to read, it's crazy good. Clear, engaging, to the point, making a difficult subject accessible without dumbing it down, no fluff or unnecessary side stories, just awesomeness.

Re: How AlphaZero Mastered Its Games

#15

Post by YeGoblynQueenne »

>> In fact, less than two months later, DeepMind published a preprint of a third paper, showing that the algorithm behind AlphaGo Zero could be generalized to any two-person, zero-sum game of perfect information (that is, a game in which there are no hidden elements, such as face-down cards in poker).

I can't find this claim in the linked paper. What I can find is a statement that AlphaZero has demonstrated that 'a general-purpose reinforcement learning algorithm can achieve, tabula rasa, superhuman performance across many challenging domains'.

Personally, and I'm sorry to be so very negative about this, but I don't even see the "many" domains. AlphaZero plays three games that are very similar to each other. Indeed, shoggi is a variant of chess. There are certainly two-person, zero-sum, perfect-information games with radically different boards and pieces to either Go, or chess and shoggi - say, the Royal Game of Ur [1], or Mancala [2], etc, not to mention stochastic games of perfect information, like backgrammon, or assymetric games like the hnefatafl games [3], and so on.

Most likely, AlphaZero can be trained to play many such games very powerfully, or at a superhuman level. The point however is that, currently, it hasn't. So no "demonstration" of general game-playing has taken place, and of course there is no such thing as some sort of theoretical analysis that would serve as proof, or indication, of such ability in any of the DeepMind papers.

I was hoping for less ra-ra cheerleading from the New Yorker, to be honest.

________________

[1] https://en.wikipedia.org/wiki/Royal_Game_of_Ur

[2] https://en.wikipedia.org/wiki/Mancala

[3] https://en.wikipedia.org/wiki/Tafl_games

Re: How AlphaZero Mastered Its Games

#16

Post by stabbles »

YeGoblynQueenne wrote: post #15

>> In fact, less than two months later, DeepMind published a preprint of a third paper, showing that the algorithm behind AlphaGo Zero could be generalized to any two-person, zero-sum game of perfect information (that is, a game in which there are no hidden elements, such as face-down cards in poker). I can't find this claim in the linked paper. What I can find is a statement that AlphaZero has demonstrated that 'a g…

I agree. There is no explanation why the neural nets work so well. There is just proof it works well for a handful specific games (Chess, Go, Shoggi). Beyond that it's just inference to the claim that neural nets work for all two-player, zero-sum, perfect information games.

However, with the framework they've built it's easy to verify the claim for new games (given sufficient computational power). Maybe that should have been the point made.

Re: How AlphaZero Mastered Its Games

#17

Post by stabbles »

ggggtez wrote: post #13
stabbles wrote: post #12

What's very interesting is that the Komodo developers have implemented a Monte Carlo Tree Search version of their engine without neural nets for evaluation / move selection. This brand new engine can actually compete at the top level (still much worse than Stockfish and slightly worse than Lc0) [1] [2] The exact implementation details are probably kept secret, but the idea is to do a few steps of minimax / alpha-beta…

You missed the point then. Alpha beta pruning requires knowledge of the game rules. Neural network pruning doesn't. The advantage is that it's a general purpose technique.

Yes, that's the main contribution of the experiment / paper. But prior to AlphaZero the chess community did not even consider investing in MCTS engines -- alpha-beta pruning was thought to be far superior. I'm thinking that we might see classical engines exploring this concept more, and maybe it's even a natural step to go from alpha-beta pruning + iterative deepening to 'best-first' search with MCTS.

Re: How AlphaZero Mastered Its Games

#18

Post by henrikf »

CJefferson wrote: post #10
glinscott wrote: post #3

James put together a really nice summary of the ideas and the projects! It was almost a year ago that lc0 was launched, since then the community (led by Alexander Lyashuk, author of the current engine) has taken it to a totally different level. Follow along at http://lczero.org ! Gcp has also done an amazing job with Leela Zero, with a very active community on the Go side. http://zero.sjeng.org Of course, DeepMind re…

I strongly suspect alphazero is easily beatable, once you have your hands on it. This is just from experience that most neural network style systems are weak against adversarial opponents who understand their internals. Of course I can't be sure, because Google refuses to give out anyone access to alphazero, or a network trained with it. Personally, that gives me more confidence they know there are significant exploi…

No need to wait for AlphaZero, you can try Leela Chess Zero today. From my experience the network without search has some blind spots, but the tree search is pretty effective in fixing them.

Re: How AlphaZero Mastered Its Games

#19

Post by bertil »

CJefferson wrote: post #10
glinscott wrote: post #3

James put together a really nice summary of the ideas and the projects! It was almost a year ago that lc0 was launched, since then the community (led by Alexander Lyashuk, author of the current engine) has taken it to a totally different level. Follow along at http://lczero.org ! Gcp has also done an amazing job with Leela Zero, with a very active community on the Go side. http://zero.sjeng.org Of course, DeepMind re…

I strongly suspect alphazero is easily beatable, once you have your hands on it. This is just from experience that most neural network style systems are weak against adversarial opponents who understand their internals. Of course I can't be sure, because Google refuses to give out anyone access to alphazero, or a network trained with it. Personally, that gives me more confidence they know there are significant exploi…

Adversarial? If the model exclusively trains against itself, you can’t really insert anything there. Do you mean, play confusing moves at the beginning of the game?

Re: How AlphaZero Mastered Its Games

#20

Post by bertil »

YeGoblynQueenne wrote: post #15

>> In fact, less than two months later, DeepMind published a preprint of a third paper, showing that the algorithm behind AlphaGo Zero could be generalized to any two-person, zero-sum game of perfect information (that is, a game in which there are no hidden elements, such as face-down cards in poker). I can't find this claim in the linked paper. What I can find is a statement that AlphaZero has demonstrated that 'a g…

This is academia, so being specific about exact claims is absolutely welcome. I would expect the problem to split between two types of games that you mention:

- asymmetric games, like hnefatafl, that probably can be covered — considering that AlphaZero can handle a late-stage situation with asymmetric options;

- what I understand to be stochastic games, of dice-based games, like the Royal Game of Ur, Mancala and backgrammon. I would expect that you have to re-define success by representing risk profile in the strategy; that could introduce complexity that the network can’t handle.

Post reply on HN