Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Fast implementation of DeepMind's AlphaZero algorithm in Julia

github.com

Post reply on HN
Print view View original

41–50 of 73 posts

Re: Fast implementation of DeepMind's AlphaZero algorithm in Julia

#41

Post by algo_trader »

cgreerrun wrote: post #5

I've been working on a Python implementation that uses Gradient Boosted Decision Trees (LightGBM/Treelite) instead of using a neural network for the value/policy models: https://github.com/cgreer/alpha-zero-boosted It's mostly to understand how AlphaZero&Friends work. I'm also curious about how well a GBDT could do, and if there are self-play techniques that can accelerate training. The nice thing about a GBDT is tha…

Its great when you can indeed iterate without 100s of GPU/hrs.

Are there any papers/comparisons/tradeoffs on when GBDT predictive-power plateaus compared to a NN?

EDIT: with self play you can trade-off a cpu budget for both the GBDT depth, a NN depth, and the roll-out depth - which is super interesting

Re: Fast implementation of DeepMind's AlphaZero algorithm in Julia

#42

Post by dklend122 »

dunefox wrote: post #29
tbenst wrote: post #10

First of all this is very cool. Dunno if author is on here, but I’m curious why both Flux and Knet are used rather than just one of them (Flux seems the most Julianic?). Also, is this really faster than PyTorch/TF? Last time I benchmarked Flux for non-trivial networks, the speed was quite good with small models but memory usage was ~5x higher than pytorch, and I couldn’t fit my models on the GPU for flux. For large m…

> 5x higher than pytorch, and I couldn’t fit my models on the GPU for flux. For large models, I had to compromise on batch size in Julia I had the exact same experience. While I like Julia and Flux I can't use it in this state for my models.

Would you mind opening corresponding issues on the repo? That would help guide the ongoing compiler work.

Re: Fast implementation of DeepMind's AlphaZero algorithm in Julia

#43

Post by vishvananda »

tromp wrote: post #30
jonath_laurent wrote: post #15

Earlier quoted context omitted.

I completely agree with you. Let me just add two remarks. First, although picking 9x9 boards makes connect-four intractable for bruteforce search indeed, I would be suprised if it made it much more difficult for AlphaZero, which relies on the generalization capabilities of the network anyway. Second, using a solved game for the tutorial is a feature, not a bug. This allows precise benchmarking of the resulting agent…

I did not see an evaluation of how close to perfection the agent becomes. Did you compute any sort of error rate (by finding moves that turn a won position into a non-won one or a drawn position into a lost one) ? And how this error rate drops over time as learning advances? That would indeed be very interesting to see.

My team did an implementation of alpha zero connect four a couple of years ago. Our findings are in a series of blog posts starting at https://medium.com/oracledevs/lessons-from-implementing-alph.... We didn't manage to get to perfection either on policy, but got pretty close. You can play against some versions of the network here: https://azfour.com

Re: Fast implementation of DeepMind's AlphaZero algorithm in Julia

#44

Post by wrsh07 »

FiberBundle wrote: post #21

Does anybody know how long it would take to train an alphazero go version using one gpu? In [1] they claim that it took 13 hours until the model was able to beat the original alphago version, but they don't state what hardware they used. [1] https://deepmind.com/blog/article/alphazero-shedding-new-lig...

That was with at least one or more tpu pods, iirc

https://cloud.google.com/tpu/docs/system-architecture

Re: Fast implementation of DeepMind's AlphaZero algorithm in Julia

#45

Post by vishvananda »

jonath_laurent wrote: post #8

Author here: I am happy to answer any question you may have about AlphaZero.jl. :-)

Nice work on this! I was behind the implementation at oracle which you referenced in the tutorial. I still keep tabs on the lc0 crowd which seems to be pushing into new ideas. Did you pull anything else from the leela crowd besides prior-temperature? It looks like maybe you also tried a WLD output head as well?

Re: Fast implementation of DeepMind's AlphaZero algorithm in Julia

#46

Post by jonath_laurent »

vishvananda wrote: post #43
tromp wrote: post #30

Earlier quoted context omitted.

I did not see an evaluation of how close to perfection the agent becomes. Did you compute any sort of error rate (by finding moves that turn a won position into a non-won one or a drawn position into a lost one) ? And how this error rate drops over time as learning advances? That would indeed be very interesting to see.

My team did an implementation of alpha zero connect four a couple of years ago. Our findings are in a series of blog posts starting at https://medium.com/oracledevs/lessons-from-implementing-alph... . We didn't manage to get to perfection either on policy, but got pretty close. You can play against some versions of the network here: https://azfour.com

Your series of blog articles has been an important source of inspiration in writing AlphaZero.jl and I cite it frequently in the documentation. Thanks to you and your team!

Re: Fast implementation of DeepMind's AlphaZero algorithm in Julia

#48

Post by jonath_laurent »

vishvananda wrote: post #45
jonath_laurent wrote: post #8

Author here: I am happy to answer any question you may have about AlphaZero.jl. :-)

Nice work on this! I was behind the implementation at oracle which you referenced in the tutorial. I still keep tabs on the lc0 crowd which seems to be pushing into new ideas. Did you pull anything else from the leela crowd besides prior-temperature? It looks like maybe you also tried a WLD output head as well?

What do you mean by WLD output head?

So far, the main idea I have pulled from the Lc0 crowd is to have a prior temperature indeed. The next thing I am planning to add is the possibility to batch inference requests across game simulations instead of relying on asynchronous MCTS. In your blog series, you anticipate the problem of the virtual loss introducing some exploration bias in the search but ultimately concludes that it does not change much:

[Citation from your blog series]: "Technically, virtual loss adds some degree of exploration to game playouts, as it forces move selection down paths that MCTS may not naturally be inclined to visit, but we never measured any detrimental (or beneficial) effect due to its use."

Interestingly, it seems that the LC0 team had a different experience here. I myself ran some tests and going from 32 to 4 workers (for 600 MCTS simulations per turn) on my connect-four agent results in a significant increase in performances. This may be due to the fact that I use a much smaller neural network than yours, which is ultimately not as strong.

Related to this, there is a question I have wanted to ask you since I found your blog article series: did you make experiments with smaller networks and what were the results? What is the smallest architecture you tried and how did it perform?

Re: Fast implementation of DeepMind's AlphaZero algorithm in Julia

#49

Post by jonath_laurent »

likeaj6 wrote: post #47

This is awesome! I worked on a similar project in the past for the game Hex Did a writeup here about it: https://notes.jasonljin.com/projects/2018/05/20/Training-Alp... https://github.com/likeaj6/alphazero-hex

Actually, I found your blog article when I was reading about AlphaZero and I found it useful!

Re: Fast implementation of DeepMind's AlphaZero algorithm in Julia

#50

Post by vishvananda »

jonath_laurent wrote: post #48
vishvananda wrote: post #45

Earlier quoted context omitted.

Nice work on this! I was behind the implementation at oracle which you referenced in the tutorial. I still keep tabs on the lc0 crowd which seems to be pushing into new ideas. Did you pull anything else from the leela crowd besides prior-temperature? It looks like maybe you also tried a WLD output head as well?

What do you mean by WLD output head? So far, the main idea I have pulled from the Lc0 crowd is to have a prior temperature indeed. The next thing I am planning to add is the possibility to batch inference requests across game simulations instead of relying on asynchronous MCTS. In your blog series, you anticipate the problem of the virtual loss introducing some exploration bias in the search but ultimately concludes…

The lc0 group has switched the result prediction to predict win, loss, and draw probabilities instead of just win/loss. Some information can be found in https://lczero.org/blog/2020/04/wdl-head/
Post reply on HN