Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Fast implementation of DeepMind's AlphaZero algorithm in Julia

github.com

Post reply on HN
Print view View original

31–40 of 73 posts

Re: Fast implementation of DeepMind's AlphaZero algorithm in Julia

#31

Post by cgreerrun »

arijun wrote: post #23
FiberBundle wrote: post #21

Does anybody know how long it would take to train an alphazero go version using one gpu? In [1] they claim that it took 13 hours until the model was able to beat the original alphago version, but they don't state what hardware they used. [1] https://deepmind.com/blog/article/alphazero-shedding-new-lig...

I can’t find it now but iirc there was a blog post on HN about a month ago that estimated their training costs at $25 million, using many TPU pods.

Here was the guestimation: https://www.yuzeh.com/data/agz-cost.html

Re: Fast implementation of DeepMind's AlphaZero algorithm in Julia

#32

Post by jonath_laurent »

dnautics wrote: post #20
jonath_laurent wrote: post #15

Earlier quoted context omitted.

I completely agree with you. Let me just add two remarks. First, although picking 9x9 boards makes connect-four intractable for bruteforce search indeed, I would be suprised if it made it much more difficult for AlphaZero, which relies on the generalization capabilities of the network anyway. Second, using a solved game for the tutorial is a feature, not a bug. This allows precise benchmarking of the resulting agent…

That's really cool and I didn't think of that. I just wanted clarification: that means you train the agent without the deterministic solution and your "validation/test" (I'm not sure what those phases are called in unsupervised learning) sets are done without the deterministic solution.

Yes, the agent is trained without access to the deterministic solution.

Re: Fast implementation of DeepMind's AlphaZero algorithm in Julia

#33

Post by jonath_laurent »

tromp wrote: post #30
jonath_laurent wrote: post #15

Earlier quoted context omitted.

I completely agree with you. Let me just add two remarks. First, although picking 9x9 boards makes connect-four intractable for bruteforce search indeed, I would be suprised if it made it much more difficult for AlphaZero, which relies on the generalization capabilities of the network anyway. Second, using a solved game for the tutorial is a feature, not a bug. This allows precise benchmarking of the resulting agent…

I did not see an evaluation of how close to perfection the agent becomes. Did you compute any sort of error rate (by finding moves that turn a won position into a non-won one or a drawn position into a lost one) ? And how this error rate drops over time as learning advances? That would indeed be very interesting to see.

Such an evaluation is available in the tutorial: https://jonathan-laurent.github.io/AlphaZero.jl/dev/tutorial...

Admittedly, the connect four agent is still far from perfect but there is a lot of margin for improvement as I have done very little hyperparameters tuning so far.

Re: Fast implementation of DeepMind's AlphaZero algorithm in Julia

#35

Post by jonath_laurent »

MaxBarraclough wrote: post #22
jonath_laurent wrote: post #8

Author here: I am happy to answer any question you may have about AlphaZero.jl. :-)

Hi, thanks for this great project. Connect Four was used as a demonstration. I presume this is because it's much easier/cheaper to train a Connect Four AI, compared to Go?

Yes. Go 19x19 would be completely intractable on a single machine (one comment is citing a $25 million cost estimate in computing power to train AlphaGo Zero). A more reasonable target would be Go 9x9 but even this would be an extreme challenge on a single machine.

There is an Oracle blog article series about training a close-to-perfect Connect Four player using AlphaZero. Even here, they had to rely on multiple GPUs.

You have to keep in mind that AlphaZero is an extremely sample-inefficient learning technique, even for simple problems. Rather, the strengths of this algorithm is that 1) it is pretty generic and 2) it can leverage huge amounts of computation.

Re: Fast implementation of DeepMind's AlphaZero algorithm in Julia

#36

Post by doublesCs »

newswasboring wrote: post #28
doublesCs wrote: post #24

Earlier quoted context omitted.

Truly truly thank you for your work <3

Not sure you are thanking jonath_laurent (original author of package of discussion) or ViralBShah (co-creator of Julia). But I concur on both accounts :D

Viral. I thanked Jonath in a different post :-P

When I see projects like this (I mean especially Julia, but also people sharing their work on packages like this) I feel very fortunate that elements of the free software movement are still alive.

Re: Fast implementation of DeepMind's AlphaZero algorithm in Julia

#37

Post by psandersen »

credit_guy wrote: post #26
psandersen wrote: post #18

Earlier quoted context omitted.

This is really interesting, thanks for sharing! I've been thinking about extensions to decision tree models that could get the benefits of NNs and it seems like there are a few ideas floating around. For example; Probabilistic Random Forests have some really interesting properties for noisy datasets, e.g. "The PRF accuracy decreased by less then 5% for a dataset with as many as 45% misclassified objects, compared to…

There appears to be a LogitBoost tree that does what you say, if I understand you correctly. [1] https://en.wikipedia.org/wiki/LogitBoost

Thanks for the reference, from a quick read it seems LogitBoost is a booster for ensembling models under a logistic loss.

I meant that the splitting point in a node in the decision trees that make up a random forest is itself a random variable that follows a distribution. Because its a smooth function (i.e. probability of splitting is 50% at the point and rises/falls smoothly) it should in principle be differentiable* so that the whole model can be trained by SGD and/or fit into an end to end learning pipeline with convolutional layers etc.

*Where I'm hazy, is how can a smooth probability function be differentiable when sampling. I'm brainstorming in the open here, will do some reading on stochastic neural networks.

Re: Fast implementation of DeepMind's AlphaZero algorithm in Julia

#38

Post by jonath_laurent »

FiberBundle wrote: post #21

Does anybody know how long it would take to train an alphazero go version using one gpu? In [1] they claim that it took 13 hours until the model was able to beat the original alphago version, but they don't state what hardware they used. [1] https://deepmind.com/blog/article/alphazero-shedding-new-lig...

I agree with the quoted numbers. As I mentioned in another comment, you have to keep in mind that AlphaZero is an extremely sample-inefficient learning technique, even for simple problems. However, it has two major strengths: 1) it is pretty generic and 2) it can leverage huge amounts of computing power.

Re: Fast implementation of DeepMind's AlphaZero algorithm in Julia

#39

Post by jonath_laurent »

patagurbon wrote: post #34
jonath_laurent wrote: post #8

Author here: I am happy to answer any question you may have about AlphaZero.jl. :-)

Do you have any thoughts about multi GPU training? I haven't seen many options for Flux previously, but didn't dig very much.

Multiple GPUs support definitely belongs to the TODO list. However, I am currently limited by the state of CUDA.jl on this, as it does not have a device-aware memory pool yet.

I am also looking forward to CUDA.jl supporting f16 and int8 computations, which may enable another big speedup.

Re: Fast implementation of DeepMind's AlphaZero algorithm in Julia

#40

Post by jonath_laurent »

doublesCs wrote: post #27
jonath_laurent wrote: post #8

Author here: I am happy to answer any question you may have about AlphaZero.jl. :-)

No questions. Just wanted to thank you for sharing. People like you make the world better one tiny bit at a time.

Thanks for your kind message.
Post reply on HN