61–70 of 91 posts
I feel like this headline is deceptive. This is a Jax reimplementation, and it was released a year ago. It is a cool library though. The basic operation of muzero is very simple, but training it efficiently is tricky.
Worth noting that while AlphaGo and AlphaZero are incredible achievements, the amount of actual code to implement them isn't very much. If you have the research paper, someone in the field could reimplement them in a few days. Then there is the large compute cost for training them to produce the trained weights. So, opensourcing these bits of work without the weights isn't as major a thing as you might imagine.
Hi I did this while I was at Google Brain and it took our team of three more like a year. The "reimplementation" part took 3 months or so and the rest of the time was literally trying to debug and figure out all of the subtleties that were not quite mentioned in the paper. See https://openreview.net/forum?id=H1eerhIpLV
Interesting approach to private variables https://github.com/deepmind/mctx/blob/577fc77a3cda1b796e277e...
I came up with a nifty implementation in Python that outperforms the naive impl by 30x, allowing a pure python MCTS/NN interop implementation. See https://www.moderndescartes.com/essays/deep_dive_mcts/
MCTX comes up with an even niftier implementation in JAX that runs the entire MCTS algorithm on the TPU. This is quite a feat because tree search is typically a heavily pointer based algorithm. It uses the object pool pattern described in https://gameprogrammingpatterns.com/object-pool.html to serialize all of the nodes of the search tree into one flat array (which is how it manages to fit into JAX formalisms). I suspect it's not a particularly efficient use of the TPU, but it does cut out all of the CPU-TPU round trip latency, which I'm sure more than compensates.
Please do the same for Alphastar, the starcraft AI, that would be great.
Worth noting that while AlphaGo and AlphaZero are incredible achievements, the amount of actual code to implement them isn't very much. If you have the research paper, someone in the field could reimplement them in a few days. Then there is the large compute cost for training them to produce the trained weights. So, opensourcing these bits of work without the weights isn't as major a thing as you might imagine.
> If you have the research paper, someone in the field could reimplement them in a few days. Hi I did this while I was at Google Brain and it took our team of three more like a year. The "reimplementation" part took 3 months or so and the rest of the time was literally trying to debug and figure out all of the subtleties that were not quite mentioned in the paper. See https://openreview.net/forum?id=H1eerhIpLV
> The replication crisis (also called the replicability crisis and the reproducibility crisis) is an ongoing methodological crisis in which the results of many scientific studies are difficult or impossible to reproduce. Because the reproducibility of empirical results is an essential part of the scientific method,[2] such failures undermine the credibility of theories building on them and potentially call into question substantial parts of scientific knowledge.
People should publish automated tests. How does a performance-optimizer know that they haven't changed the output of there are no known-good inputs and outputs documented as executable tests? Pytest-hypothesis seems like a nice compact way to specify tests.
AlphaZero: https://en.wikipedia.org/wiki/AlphaZero
GH topic "AlphaZero" https://github.com/topics/alphazero
I believe ther are one or more JAX implementations of AlphaZero?
Though there's not yet a quantum-inference-based self-play (AlphaZero) algorithm?
TIL about the modified snow plow problem is a variation on TSP, and there are already quantum algos capable of optimally solving TSP.
So the naive MCTS implementation in Python is ridiculously inefficient. Of course, you could reimplement it in C++ but this then requires you to use the C wrappers of Tensorflow/JAX to do the MCTS/neural network interop. I came up with a nifty implementation in Python that outperforms the naive impl by 30x, allowing a pure python MCTS/NN interop implementation. See https://www.moderndescartes.com/essays/deep_dive_mct…
Great post!
Chasing pointers in the MCTS tree is definitely a slow approach. Although typically there are ~ 900 "considerations" per move for alphazero. I've found getting value/policy predictions from a neural network (or GBDT[1]) for the node expansions during those considerations is at least an order of magnitude slower than the MCTS tree-hopping logic.
Earlier quoted context omitted.
Yep, there are many reimplementations. Here is a reimplementation that swaps out a neural net with a GBDT to address compute costs: https://github.com/cgreer/alpha-zero-boosted
How does the performance of this version compare?
High level, I'd say it's a good way to test a new environment w/out spending time/effort on GPUs until you understand the problem well, and then you can switch to the time/money costly GPU world.
Worth noting that while AlphaGo and AlphaZero are incredible achievements, the amount of actual code to implement them isn't very much. If you have the research paper, someone in the field could reimplement them in a few days. Then there is the large compute cost for training them to produce the trained weights. So, opensourcing these bits of work without the weights isn't as major a thing as you might imagine.
If only there existed a distributed way to incentivize calculation of AI weightings while also providing a currency to encourage scale…
Worth noting that while AlphaGo and AlphaZero are incredible achievements, the amount of actual code to implement them isn't very much. If you have the research paper, someone in the field could reimplement them in a few days. Then there is the large compute cost for training them to produce the trained weights. So, opensourcing these bits of work without the weights isn't as major a thing as you might imagine.
There is Leela Zero ( https://github.com/leela-zero/leela-zero ) for Go and lc0/Leela Chess ( https://github.com/orgs/LeelaChessZero/repositories ) for Chess, where both provide trained weights. The Leela Chess project specifically have been working for a long time on training and refining the weights for Chess, as well as providing the code -- they allow you to see the history and performance over time for the vario…