Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

MuZero: Mastering Go, chess, shogi and Atari without rules

deepmind.com

Post reply on HN
Print view View original

31–40 of 79 posts

Re: MuZero: Mastering Go, chess, shogi and Atari without rules

#31

Post by cgreerrun »

panabee wrote: post #25
cgreerrun wrote: post #15

Earlier quoted context omitted.

What you can do is checkout the algorithm at particular stages of development. AlphaZero&Friends start out not being very good at the game, then over time they learn and eventually become super human. You typically checkpoint the weights for the model at various stages. So early on, the algo would be like a 600 elo player for chess and then eventually get to superhuman elo levels. If you wanted to train using an Alph…

thanks for sharing! this is very interesting. why did you use GBDTs instead of NNs?

> Thanks for sharing!

You're welcome.

> Why did you use GBDTs instead of NNs?

I mostly wanted to build an implementation to see how it worked; I was more familiar with GBDTs than NNs, so I figured I'd start with that. At its heart, AlphaZero is the marriage of two great ideas: using a Monte Carlo Tree Search (MCTS) to efficiently look ahead and find good moves and using a powerful ML model (like a ResNet) as a bot's intuition about which positions are good to be in (value network) and which moves are good when you're in which positions (policy network). So if a GBDT is powerful enough for your use case, the "ML Model" component in the MCTS+ML Model AlphaZero setup should be able to be swapped out with it if you want.

But I was also curious if GBDTs would do almost as well as a NN, because GBDTs can be much more efficient w.r.t. cost/energy. At the time when AlphaZero came out, I think it cost >$10M to train a superhuman Go algo. Nowadays KatoGo [1] can do it for Check out this HN thread if you're interested in more AlphaX shenanigans: https://news.ycombinator.com/item?id=23599278

[1] https://github.com/lightvector/KataGo

Re: MuZero: Mastering Go, chess, shogi and Atari without rules

#32

Post by ska »

dwaltrip wrote: post #30
ska wrote: post #24

Earlier quoted context omitted.

It's still not that obvious. There has been a lot of interesting stuff in this current iteration of "AI", but the overall approach could still end up being a dead end with respect to AGI itself. It's an old discussion, and while a few of the deep learning results are really impressive I don't think any of them have fundamentally changed that discussion, yet.

Even if it is a dead end, that is still valuable knowledge.

Of course, but the claim wasn't that it had no value.

You can easily handwave that all generated knowledge might be indirectly useful; i think that's fair but also different than the distinction I drew.

Re: MuZero: Mastering Go, chess, shogi and Atari without rules

#33

Post by chongli »

ignoranceprior wrote: post #3

Whoa, this is extremely impressive. Quotes from the BBC article: > "For the first time, we actually have a system which is able to build its own understanding of how the world works, and use that understanding to do this kind of sophisticated look-ahead planning that you've previously seen for games like chess. > "[It] can start from nothing, and just through trial and error both discover the rules of the world and u…

What does it mean to “not be given the rules”? If you set a child down in front of a chess board with the pieces nearby and they are not aware of the rules, I doubt they’d ever figure out how to play even a single correct game of chess. Heck, the child may decide to put the pieces in their mouth or dress them up as make belief characters.

Without any concept of the rules you have no way of even knowing that you’ve set up the pieces for a legal starting position, never mind executing a legal move to open the game.

This is really bizarre.

Re: MuZero: Mastering Go, chess, shogi and Atari without rules

#34

Post by ignoranceprior »

chongli wrote: post #33
ignoranceprior wrote: post #3

Whoa, this is extremely impressive. Quotes from the BBC article: > "For the first time, we actually have a system which is able to build its own understanding of how the world works, and use that understanding to do this kind of sophisticated look-ahead planning that you've previously seen for games like chess. > "[It] can start from nothing, and just through trial and error both discover the rules of the world and u…

What does it mean to “not be given the rules”? If you set a child down in front of a chess board with the pieces nearby and they are not aware of the rules, I doubt they’d ever figure out how to play even a single correct game of chess. Heck, the child may decide to put the pieces in their mouth or dress them up as make belief characters. Without any concept of the rules you have no way of even knowing that you’ve se…

This is explained in Appendix A of the paper ("Comparison to AlphaZero"): https://arxiv.org/pdf/1911.08265.pdf

Basically, AlphaZero was provided with a simulator that was able to distinguish legal and illegal moves and determine which future game states would be wins or losses. This was used to generate the search tree of possible states and actions.

MuZero doesn't have access to a simulator, it only has access to its direct environment. MuZero excludes actions that are immediately illegal, which solves the problem you mention in your penultimate paragraph, but it needs to learn the game's dynamics in order to determine which future moves and states are possible.

Re: MuZero: Mastering Go, chess, shogi and Atari without rules

#37

Post by syntheticmindai »

mark_l_watson wrote: post #36

I watched the Alpha Go vs. Lee Sedol games live. Big fan. That said, I think Deep Mind should go all in for solving practical real world problems.

They have used muzero to do video compression and saved 5% of bits. Source: david silver wired.co.uk interview

Re: MuZero: Mastering Go, chess, shogi and Atari without rules

#38

Post by panabee »

cgreerrun wrote: post #31
panabee wrote: post #25

Earlier quoted context omitted.

thanks for sharing! this is very interesting. why did you use GBDTs instead of NNs?

> Thanks for sharing! You're welcome. > Why did you use GBDTs instead of NNs? I mostly wanted to build an implementation to see how it worked; I was more familiar with GBDTs than NNs, so I figured I'd start with that. At its heart, AlphaZero is the marriage of two great ideas: using a Monte Carlo Tree Search (MCTS) to efficiently look ahead and find good moves and using a powerful ML model (like a ResNet) as a bot's…

will do, thanks for sharing this other HN thread. also sent an email to the address listed on your github profile.

Re: MuZero: Mastering Go, chess, shogi and Atari without rules

#39

Post by syntaxing »

As impractical as the idea is, reinforcement learning is so damn fun. I highly recommend others to play around with it. I originally was using the famous fork of OpenAi baseline, stable baseline but had issues with tuning with Optuna. I recently stumbled across Ray from Berkley [1] and it has a newer and fancier built-in hyper-parameter tuner. Even as a hardware engineer that's only a software hobbyist can make the computer play some atari games. I think my next step is to try to make my own Super Mario agent.

[1] https://docs.ray.io/en/latest/index.html

Re: MuZero: Mastering Go, chess, shogi and Atari without rules

#40

Post by webmaven »

I may be missing something, but it seems that what is being described is a neural net architecture that can be trained on any of several games to get impressive results for those game.

NOT that one neural net can be trained to play all of the games.

So, while this is an interesting result and makes using the same architecture for specific applications easier and a bit more plug-and-play with little to no modification of the code, what it accomplishes is reducing the effort required by a software engineer or researcher on adapting the software before training even begins, but it pretty much still requires the same amount of training.

What this doesn't seem to do is allow the same trained network to be applied to multiple tasks (which I think most of the AGI comments are assuming), and it certainly doesn't generalize anything among the games it is trained on.

Post reply on HN