Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Fast implementation of DeepMind's AlphaZero algorithm in Julia

github.com

Post reply on HN
Print view View original

11–20 of 73 posts

Re: Fast implementation of DeepMind's AlphaZero algorithm in Julia

#11

Post by jonath_laurent »

cgreerrun wrote: post #5

I've been working on a Python implementation that uses Gradient Boosted Decision Trees (LightGBM/Treelite) instead of using a neural network for the value/policy models: https://github.com/cgreer/alpha-zero-boosted It's mostly to understand how AlphaZero&Friends work. I'm also curious about how well a GBDT could do, and if there are self-play techniques that can accelerate training. The nice thing about a GBDT is tha…

This is very interesting! If your experiments work out, I would be interested in adding "Gradient Boosted Decision Trees" support to AlphaZero.jl.

I saw the work on KataGo and implementing "Playout Cap Randomization" is indeed on my TODO list.

Re: Fast implementation of DeepMind's AlphaZero algorithm in Julia

#12

Post by jonath_laurent »

tbenst wrote: post #10

First of all this is very cool. Dunno if author is on here, but I’m curious why both Flux and Knet are used rather than just one of them (Flux seems the most Julianic?). Also, is this really faster than PyTorch/TF? Last time I benchmarked Flux for non-trivial networks, the speed was quite good with small models but memory usage was ~5x higher than pytorch, and I couldn’t fit my models on the GPU for flux. For large m…

Author here. AlphaZero.jl supports both Flux and Knet indeed and users can choose whatever framework they want to use.

As far as I understand, Flux and Knet have different strengths. I think Knet is a bit more stable and mature for large-scale Deep Learning, but Flux shines for "scientific-ML" usecases where low AD overhead is crucial.

Re: Fast implementation of DeepMind's AlphaZero algorithm in Julia

#13

Post by jonath_laurent »

tbenst wrote: post #10

First of all this is very cool. Dunno if author is on here, but I’m curious why both Flux and Knet are used rather than just one of them (Flux seems the most Julianic?). Also, is this really faster than PyTorch/TF? Last time I benchmarked Flux for non-trivial networks, the speed was quite good with small models but memory usage was ~5x higher than pytorch, and I couldn’t fit my models on the GPU for flux. For large m…

I suspect FLux/Knet are still slightly slower and less memory efficient than PyTorch/TF, although things are moving very fast here!

This is not relevant in understanding AlphaZero.jl speed though. The reason it is much faster than Python implementations is because tree search is also a bottleneck, and Julia shines here!

Re: Fast implementation of DeepMind's AlphaZero algorithm in Julia

#14

Post by tromp »

The implementation includes Connect Four as an example application. While the standard board size of 7x6 is indeed solved, as they note, and in fact all sizes up to 8x8 are [1], they could have picked 9x8 or 9x9 which are currently unsolved. The latter is the new standard size on Little Golem which upgraded from 8x8 when that was solved.

[1] https://tromp.github.io/c4/c4.html

[2] http://www.littlegolem.net/jsp/games/gamedetail.jsp? gtid=fir

[3] http://www.littlegolem.net/jsp/forum/topic2.jsp?forum=80&top...

Re: Fast implementation of DeepMind's AlphaZero algorithm in Julia

#15

Post by jonath_laurent »

tromp wrote: post #14

The implementation includes Connect Four as an example application. While the standard board size of 7x6 is indeed solved, as they note, and in fact all sizes up to 8x8 are [1], they could have picked 9x8 or 9x9 which are currently unsolved. The latter is the new standard size on Little Golem which upgraded from 8x8 when that was solved. [1] https://tromp.github.io/c4/c4.html [2] http://www.littlegolem.net/jsp/games/…

I completely agree with you. Let me just add two remarks. First, although picking 9x9 boards makes connect-four intractable for bruteforce search indeed, I would be suprised if it made it much more difficult for AlphaZero, which relies on the generalization capabilities of the network anyway. Second, using a solved game for the tutorial is a feature, not a bug. This allows precise benchmarking of the resulting agent as a ground truth is known.

Re: Fast implementation of DeepMind's AlphaZero algorithm in Julia

#16

Post by tbenst »

jonath_laurent wrote: post #13
tbenst wrote: post #10

First of all this is very cool. Dunno if author is on here, but I’m curious why both Flux and Knet are used rather than just one of them (Flux seems the most Julianic?). Also, is this really faster than PyTorch/TF? Last time I benchmarked Flux for non-trivial networks, the speed was quite good with small models but memory usage was ~5x higher than pytorch, and I couldn’t fit my models on the GPU for flux. For large m…

I suspect FLux/Knet are still slightly slower and less memory efficient than PyTorch/TF, although things are moving very fast here! This is not relevant in understanding AlphaZero.jl speed though. The reason it is much faster than Python implementations is because tree search is also a bottleneck, and Julia shines here!

Ah, I hadn’t appreciated this. Thanks for making & sharing your code!

Re: Fast implementation of DeepMind's AlphaZero algorithm in Julia

#17

Post by newswasboring »

ViralBShah wrote: post #6
mtgp1000 wrote: post #3

I don't know anything about Julia...how hard would this be to port to python or a c-style language? Edit: I was mainly asking because I was curious about the relative expressiveness Julia...

I was going through this project over the weekend. And while I can't recall where exactly in the docs I read this, I am quite sure the author mentioned that there are various python projects but they are quite slow. Other implementations such as leela chess zero have a lot of C++ and are difficult to follow. In fact, one of the things we want to do is maximize the performance of the Julia implementation. We hope to c…

I don't know if you saw it here, but a similar point is made in the readme section "Why should I care about this implementation".[1]

https://github.com/jonathan-laurent/AlphaZero.jl#why-should-...

Re: Fast implementation of DeepMind's AlphaZero algorithm in Julia

#18

Post by psandersen »

cgreerrun wrote: post #5

I've been working on a Python implementation that uses Gradient Boosted Decision Trees (LightGBM/Treelite) instead of using a neural network for the value/policy models: https://github.com/cgreer/alpha-zero-boosted It's mostly to understand how AlphaZero&Friends work. I'm also curious about how well a GBDT could do, and if there are self-play techniques that can accelerate training. The nice thing about a GBDT is tha…

This is really interesting, thanks for sharing!

I've been thinking about extensions to decision tree models that could get the benefits of NNs and it seems like there are a few ideas floating around.

For example; Probabilistic Random Forests have some really interesting properties for noisy datasets, e.g. "The PRF accuracy decreased by less then 5% for a dataset with as many as 45% misclassified objects, compared to a clean dataset." - https://arxiv.org/abs/1811.05994

PRF's might be a natural fit for RL, especially methods using monte carlo tree search.

Speculating here as I'm not adequately familiar with stochastic calculus, but intuitively it seems like probabilistic decision trees could be made differentiable since the hard decision threshold in a tree could be a turned continuous (i.e. every split is logistic regression), which might enable some really interesting applications. I personally dream of being able to cleanly integrate decision trees and tools from NNs in something like pyro for a fully Bayesian model.

Re: Fast implementation of DeepMind's AlphaZero algorithm in Julia

#19

Post by ViralBShah »

tbenst wrote: post #10

First of all this is very cool. Dunno if author is on here, but I’m curious why both Flux and Knet are used rather than just one of them (Flux seems the most Julianic?). Also, is this really faster than PyTorch/TF? Last time I benchmarked Flux for non-trivial networks, the speed was quite good with small models but memory usage was ~5x higher than pytorch, and I couldn’t fit my models on the GPU for flux. For large m…

While some may be addressed and others are being addressed, what would really help us if people file issues when they don't find performance to be adequate. If you still have the code handy, please do open some issues.

Re: Fast implementation of DeepMind's AlphaZero algorithm in Julia

#20

Post by dnautics »

jonath_laurent wrote: post #15
tromp wrote: post #14

The implementation includes Connect Four as an example application. While the standard board size of 7x6 is indeed solved, as they note, and in fact all sizes up to 8x8 are [1], they could have picked 9x8 or 9x9 which are currently unsolved. The latter is the new standard size on Little Golem which upgraded from 8x8 when that was solved. [1] https://tromp.github.io/c4/c4.html [2] http://www.littlegolem.net/jsp/games/…

I completely agree with you. Let me just add two remarks. First, although picking 9x9 boards makes connect-four intractable for bruteforce search indeed, I would be suprised if it made it much more difficult for AlphaZero, which relies on the generalization capabilities of the network anyway. Second, using a solved game for the tutorial is a feature, not a bug. This allows precise benchmarking of the resulting agent…

That's really cool and I didn't think of that. I just wanted clarification: that means you train the agent without the deterministic solution and your "validation/test" (I'm not sure what those phases are called in unsupervised learning) sets are done without the deterministic solution.
Post reply on HN