Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Fast implementation of DeepMind's AlphaZero algorithm in Julia

github.com

Post reply on HN
Print view View original

61–70 of 73 posts

Re: Fast implementation of DeepMind's AlphaZero algorithm in Julia

#61

Post by gbrown »

cgreerrun wrote: post #58
psandersen wrote: post #18

Earlier quoted context omitted.

This is really interesting, thanks for sharing! I've been thinking about extensions to decision tree models that could get the benefits of NNs and it seems like there are a few ideas floating around. For example; Probabilistic Random Forests have some really interesting properties for noisy datasets, e.g. "The PRF accuracy decreased by less then 5% for a dataset with as many as 45% misclassified objects, compared to…

A fast, powerful bayesian model seems like it would be a game-changer. PUCT (the heart of the AlphaZero MCTS that decides which action to choose) really seems setup to model the action choices as a multinomial bayesian inference problem (it already updates the action priors with Dir noise). Thanks for the link! I don't really know anything about the world of probabilistic trees. I'll check it out. The only bayesian a…

Check out the SoftBART method, I think there are interesting optimizations possible there. XBart also looks promising as an approach.

Re: Fast implementation of DeepMind's AlphaZero algorithm in Julia

#62

Post by jonath_laurent »

cgreerrun wrote: post #60
algo_trader wrote: post #41

Earlier quoted context omitted.

Its great when you can indeed iterate without 100s of GPU/hrs. Are there any papers/comparisons/tradeoffs on when GBDT predictive-power plateaus compared to a NN? EDIT: with self play you can trade-off a cpu budget for both the GBDT depth, a NN depth, and the roll-out depth - which is super interesting

> Are there any papers/comparisons/tradeoffs on when GBDT predictive-power plateaus compared to a NN? None specifically that I know of, but I haven't searched. "Shallow learning" GBDTs can do pretty well on MNIST ( https://www.kaggle.com/c/digit-recognizer/discussion/61480 ), getting 98%+ accuracy compared to the 99%+ of NNs. So I figured if they can handle MNIST, they can probably handle connect 4, and would be usef…

I am wondering if your idea of using GBDTs in combination with AlphaZero might not be most influential in areas where no neural network architecture is known to provide the right inductive bias for the problem at hand.

I think neural models are pretty unbeatable in many classic RL environments because convolutional neural networks are REALLY good at learning visual representations. In some sense, I suspect that the great success of AlphaGo Zero comes in big part from the fact that it really makes sense to analyze a Go board as a 2D image using convolutional networks: convolutional networks provide the right inductive bias for the problem of learning to play Go.

However, there are tasks where neural network are not as good, such as symbolic manipulation tasks (I am in a good position to know this as I'm doing research in the area of automated theorem proving). I would be very curious to see how your approach fares for those tasks.

Re: Fast implementation of DeepMind's AlphaZero algorithm in Julia

#63

Post by cgreerrun »

jorgemf wrote: post #7
cgreerrun wrote: post #5

I've been working on a Python implementation that uses Gradient Boosted Decision Trees (LightGBM/Treelite) instead of using a neural network for the value/policy models: https://github.com/cgreer/alpha-zero-boosted It's mostly to understand how AlphaZero&Friends work. I'm also curious about how well a GBDT could do, and if there are self-play techniques that can accelerate training. The nice thing about a GBDT is tha…

how good is your AI so far?

I'm trying to answer that right now, actually.

For connect 4, once it's trained a bit it seems to do really well. At 800 MCTS playouts (Connect 4 is a solved game, so it should be possible to sample the space of the trillions of (position, who should win?, what are the best move(s)?) tuples and compare the answers to your value/policy models to get some kind of objective error. I haven't had time to do that, but having that benchmark is nice to have so you don't have to do a "ladder tournament" against some reference bot(s) like you do for Go where you don't know what ideal play is.

After training it for 10 hours on Quoridor (using my personal laptop), it still can't beat me, but it doesn't seem anywhere close to plateauing. It goes from the agents aimlessly wandering around the board looking for victory row and randomly placing walls, to putting walls that thwart the opponent and navigating to the victory row.

I decided to implement PCR and try out some self-play techniques on Connect Four before I give it another go for Quoridor; a few days of self-play improvements can speedup training 10x. That's where I'm at now...

Once I test a few strategies I was thinking of firing up a c5a24x, 96-core box on AWS and giving it another go. It's ~1-2$/hr at the spot price so I can probably do a lot of damage for 50$ or so.

Re: Fast implementation of DeepMind's AlphaZero algorithm in Julia

#64

Post by [deleted] »

jonath_laurent wrote: post #56
master_yoda_1 wrote: post #55

Earlier quoted context omitted.

I am confused about the FAST part, it is faster than all the other implementation (some of them are in c++) or it is just julia implementation and you think it is fast? I am asking because if julia is faster than c++ for ml/dl I would prefer to use it for production use cases.

This needs clarification indeed. As I explain in the documentation, the aim of AlphaZero.jl is not to compete with hyper-specialized and hyper-optimized implementations such as LC0 or ELF OpenGO. These implementations are written in C++ with custom CUDA kernels and they are optimized for highly distributed computing environments. They are also very complex and therefore pretty inaccessible to students and researchers…

[deleted]

Re: Fast implementation of DeepMind's AlphaZero algorithm in Julia

#65

Post by dklend122 »

master_yoda_1 wrote: post #55
jonath_laurent wrote: post #8

Author here: I am happy to answer any question you may have about AlphaZero.jl. :-)

I am confused about the FAST part, it is faster than all the other implementation (some of them are in c++) or it is just julia implementation and you think it is fast? I am asking because if julia is faster than c++ for ml/dl I would prefer to use it for production use cases.

In addition to the excellent answer by the author below, I'd like to say that Julia can get within spitting distance (or even sometimes exceed) C++ speeds (and even BLAS). So if a comparable amount of work went into optimizing specific paths through generic code (or even the generic code itself), it could be as fast. Also, one can write CUDA kernels in pure Julia.

Re: Fast implementation of DeepMind's AlphaZero algorithm in Julia

#66

Post by O_H_E »

jonath_laurent wrote: post #56
master_yoda_1 wrote: post #55

Earlier quoted context omitted.

I am confused about the FAST part, it is faster than all the other implementation (some of them are in c++) or it is just julia implementation and you think it is fast? I am asking because if julia is faster than c++ for ml/dl I would prefer to use it for production use cases.

This needs clarification indeed. As I explain in the documentation, the aim of AlphaZero.jl is not to compete with hyper-specialized and hyper-optimized implementations such as LC0 or ELF OpenGO. These implementations are written in C++ with custom CUDA kernels and they are optimized for highly distributed computing environments. They are also very complex and therefore pretty inaccessible to students and researchers…

Always great to see someone who finds something broken, and fixes it for others to move forward.

Thank you.

Re: Fast implementation of DeepMind's AlphaZero algorithm in Julia

#67

Post by master_yoda_1 »

dklend122 wrote: post #65
master_yoda_1 wrote: post #55

Earlier quoted context omitted.

I am confused about the FAST part, it is faster than all the other implementation (some of them are in c++) or it is just julia implementation and you think it is fast? I am asking because if julia is faster than c++ for ml/dl I would prefer to use it for production use cases.

In addition to the excellent answer by the author below, I'd like to say that Julia can get within spitting distance (or even sometimes exceed) C++ speeds (and even BLAS). So if a comparable amount of work went into optimizing specific paths through generic code (or even the generic code itself), it could be as fast. Also, one can write CUDA kernels in pure Julia.

Looks like I need to start learning julia

Re: Fast implementation of DeepMind's AlphaZero algorithm in Julia

#69

Post by klipt »

jonath_laurent wrote: post #38
FiberBundle wrote: post #21

Does anybody know how long it would take to train an alphazero go version using one gpu? In [1] they claim that it took 13 hours until the model was able to beat the original alphago version, but they don't state what hardware they used. [1] https://deepmind.com/blog/article/alphazero-shedding-new-lig...

I agree with the quoted numbers. As I mentioned in another comment, you have to keep in mind that AlphaZero is an extremely sample-inefficient learning technique, even for simple problems. However, it has two major strengths: 1) it is pretty generic and 2) it can leverage huge amounts of computing power.

What would be an example of a more sample efficient algorithm?
Post reply on HN