Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Fast implementation of DeepMind's AlphaZero algorithm in Julia

github.com

Post reply on HN
Print view View original

51–60 of 73 posts

Re: Fast implementation of DeepMind's AlphaZero algorithm in Julia

#51

Post by vishvananda »

jonath_laurent wrote: post #48
vishvananda wrote: post #45

Earlier quoted context omitted.

Nice work on this! I was behind the implementation at oracle which you referenced in the tutorial. I still keep tabs on the lc0 crowd which seems to be pushing into new ideas. Did you pull anything else from the leela crowd besides prior-temperature? It looks like maybe you also tried a WLD output head as well?

What do you mean by WLD output head? So far, the main idea I have pulled from the Lc0 crowd is to have a prior temperature indeed. The next thing I am planning to add is the possibility to batch inference requests across game simulations instead of relying on asynchronous MCTS. In your blog series, you anticipate the problem of the virtual loss introducing some exploration bias in the search but ultimately concludes…

we did a lot of our early experimentation with small networks. I don't think we went any smaller than 5 layers of 64 filters as we mentioned here: https://medium.com/oracledevs/lessons-from-alpha-zero-part-5...

Re: Fast implementation of DeepMind's AlphaZero algorithm in Julia

#52

Post by jonath_laurent »

vishvananda wrote: post #51
jonath_laurent wrote: post #48

Earlier quoted context omitted.

What do you mean by WLD output head? So far, the main idea I have pulled from the Lc0 crowd is to have a prior temperature indeed. The next thing I am planning to add is the possibility to batch inference requests across game simulations instead of relying on asynchronous MCTS. In your blog series, you anticipate the problem of the virtual loss introducing some exploration bias in the search but ultimately concludes…

we did a lot of our early experimentation with small networks. I don't think we went any smaller than 5 layers of 64 filters as we mentioned here: https://medium.com/oracledevs/lessons-from-alpha-zero-part-5...

And what were the results of these experiments? What error rate can you reach with the smallest network architecture you tried for example?

Re: Fast implementation of DeepMind's AlphaZero algorithm in Julia

#53

Post by vishvananda »

jonath_laurent wrote: post #52
vishvananda wrote: post #51

Earlier quoted context omitted.

we did a lot of our early experimentation with small networks. I don't think we went any smaller than 5 layers of 64 filters as we mentioned here: https://medium.com/oracledevs/lessons-from-alpha-zero-part-5...

And what were the results of these experiments? What error rate can you reach with the smallest network architecture you tried for example?

Unfortunately I don't remember the exact numbers, but I think it was a couple percentage points worse than we were able to get with the large models.

Re: Fast implementation of DeepMind's AlphaZero algorithm in Julia

#54

Post by jonath_laurent »

vishvananda wrote: post #53
jonath_laurent wrote: post #52

Earlier quoted context omitted.

And what were the results of these experiments? What error rate can you reach with the smallest network architecture you tried for example?

Unfortunately I don't remember the exact numbers, but I think it was a couple percentage points worse than we were able to get with the large models.

This is interesting, thanks! Is there anything else you can tell me about the results of your experiments with small networks? I am really interested in this.

For example: did you notice than increasing or decreasing network size required significant changes in other hyperparameters? Are small networks learning faster at the beginning of training before they start to plateau?

Re: Fast implementation of DeepMind's AlphaZero algorithm in Julia

#55

Post by master_yoda_1 »

jonath_laurent wrote: post #8

Author here: I am happy to answer any question you may have about AlphaZero.jl. :-)

I am confused about the FAST part, it is faster than all the other implementation (some of them are in c++) or it is just julia implementation and you think it is fast? I am asking because if julia is faster than c++ for ml/dl I would prefer to use it for production use cases.

Re: Fast implementation of DeepMind's AlphaZero algorithm in Julia

#56

Post by jonath_laurent »

master_yoda_1 wrote: post #55
jonath_laurent wrote: post #8

Author here: I am happy to answer any question you may have about AlphaZero.jl. :-)

I am confused about the FAST part, it is faster than all the other implementation (some of them are in c++) or it is just julia implementation and you think it is fast? I am asking because if julia is faster than c++ for ml/dl I would prefer to use it for production use cases.

This needs clarification indeed. As I explain in the documentation, the aim of AlphaZero.jl is not to compete with hyper-specialized and hyper-optimized implementations such as LC0 or ELF OpenGO. These implementations are written in C++ with custom CUDA kernels and they are optimized for highly distributed computing environments. They are also very complex and therefore pretty inaccessible to students and researchers.

The philosophy of AlphaZero.jl is to provide an implementation of AlphaZero that is simple enough to be widely accessible for students and researchers, while also being sufficiently powerful and fast to enable meaningful experiments on limited computing resources. It has the simplicity of the many existing python implementations, while being consistently between one and two orders of magnitude faster.

More generally, the AlphaZero algorithm is extremely general and I think it can find applications in many research domains (including automated theorem proving, which is my own research area). I have been surprised to see that, despite the general excitement around AlphaZero, very few people actually tried to build on it. One explanation, I think, is the lack of accessible open-source implementations. I am trying to bridge this gap with AlphaZero.jl.

Re: Fast implementation of DeepMind's AlphaZero algorithm in Julia

#57

Post by cgreerrun »

jonath_laurent wrote: post #11
cgreerrun wrote: post #5

I've been working on a Python implementation that uses Gradient Boosted Decision Trees (LightGBM/Treelite) instead of using a neural network for the value/policy models: https://github.com/cgreer/alpha-zero-boosted It's mostly to understand how AlphaZero&Friends work. I'm also curious about how well a GBDT could do, and if there are self-play techniques that can accelerate training. The nice thing about a GBDT is tha…

This is very interesting! If your experiments work out, I would be interested in adding "Gradient Boosted Decision Trees" support to AlphaZero.jl. I saw the work on KataGo and implementing "Playout Cap Randomization" is indeed on my TODO list.

I'll be sure to let you know how it goes. I have a few self-play experiments that I'm trying out once I get the testing setup finished.

Do you know if there is a discord channel for all of us AlphaZero nerds that I'm missing? If not, we should make one. It'd be great to bounce ideas off each other.

Re: Fast implementation of DeepMind's AlphaZero algorithm in Julia

#58

Post by cgreerrun »

psandersen wrote: post #18
cgreerrun wrote: post #5

I've been working on a Python implementation that uses Gradient Boosted Decision Trees (LightGBM/Treelite) instead of using a neural network for the value/policy models: https://github.com/cgreer/alpha-zero-boosted It's mostly to understand how AlphaZero&Friends work. I'm also curious about how well a GBDT could do, and if there are self-play techniques that can accelerate training. The nice thing about a GBDT is tha…

This is really interesting, thanks for sharing! I've been thinking about extensions to decision tree models that could get the benefits of NNs and it seems like there are a few ideas floating around. For example; Probabilistic Random Forests have some really interesting properties for noisy datasets, e.g. "The PRF accuracy decreased by less then 5% for a dataset with as many as 45% misclassified objects, compared to…

A fast, powerful bayesian model seems like it would be a game-changer. PUCT (the heart of the AlphaZero MCTS that decides which action to choose) really seems setup to model the action choices as a multinomial bayesian inference problem (it already updates the action priors with Dir noise).

Thanks for the link! I don't really know anything about the world of probabilistic trees. I'll check it out.

The only bayesian approach to decision trees I'm familiar with is BART (https://projecteuclid.org/download/pdfview_1/euclid.aoas/127...). I haven't used them, but I'm guessing because it uses MCMC to update the params it's not super fast. I've seen them used in causality applications for partial dependency plots where it's convenient to convey the certainty of a variable's effect.

Re: Fast implementation of DeepMind's AlphaZero algorithm in Julia

#59

Post by jonath_laurent »

cgreerrun wrote: post #57
jonath_laurent wrote: post #11

Earlier quoted context omitted.

This is very interesting! If your experiments work out, I would be interested in adding "Gradient Boosted Decision Trees" support to AlphaZero.jl. I saw the work on KataGo and implementing "Playout Cap Randomization" is indeed on my TODO list.

I'll be sure to let you know how it goes. I have a few self-play experiments that I'm trying out once I get the testing setup finished. Do you know if there is a discord channel for all of us AlphaZero nerds that I'm missing? If not, we should make one. It'd be great to bounce ideas off each other.

Thanks! You can easily find my email address from my github. I am not aware of a canonical discord channel for discussing AlphaZero but the Lc0 community has a discord channel on which I had some interesting conversations.

Re: Fast implementation of DeepMind's AlphaZero algorithm in Julia

#60

Post by cgreerrun »

algo_trader wrote: post #41
cgreerrun wrote: post #5

I've been working on a Python implementation that uses Gradient Boosted Decision Trees (LightGBM/Treelite) instead of using a neural network for the value/policy models: https://github.com/cgreer/alpha-zero-boosted It's mostly to understand how AlphaZero&Friends work. I'm also curious about how well a GBDT could do, and if there are self-play techniques that can accelerate training. The nice thing about a GBDT is tha…

Its great when you can indeed iterate without 100s of GPU/hrs. Are there any papers/comparisons/tradeoffs on when GBDT predictive-power plateaus compared to a NN? EDIT: with self play you can trade-off a cpu budget for both the GBDT depth, a NN depth, and the roll-out depth - which is super interesting

> Are there any papers/comparisons/tradeoffs on when GBDT predictive-power plateaus compared to a NN?

None specifically that I know of, but I haven't searched.

"Shallow learning" GBDTs can do pretty well on MNIST (https://www.kaggle.com/c/digit-recognizer/discussion/61480), getting 98%+ accuracy compared to the 99%+ of NNs. So I figured if they can handle MNIST, they can probably handle connect 4, and would be useful to explore the self-play training efficiency aspects of AlphaZero (at orders of magnitude less compute time/cost)

> with self play you can trade-off a cpu budget for both the GBDT depth, a NN depth, and the roll-out depth - which is super interesting.

Definitely. It'll be interesting to see if a deeper MCTS search with a less powerful model can do pretty well. I'm still fairly ignorant about the MCTS literature, but I've definitely seen MCTS married to other value/policy models (linear regressions, for e.g.) that used large numbers of playouts years before Alpha Go came out. Those didn't work out, so seems like the DL aspect of Alpha Zero is somewhat essential to be able to learn games as complex as Go.

Post reply on HN