Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Mastering Chess and Shogi by Self-Play with General Reinforcement Learning

arxiv.org

Post reply on HN
Print view View original

141–150 of 282 posts

Re: Mastering Chess and Shogi by Self-Play with General Reinforcement Learning

#141

Post by visarga »

backpropaganda wrote: post #137
visarga wrote: post #133

Earlier quoted context omitted.

It's easy for many to think that solving Go and chess means we can also solve household work like cleaning, cooking and washing dishes but it's actually harder.

Next up: Google's Deepmind AI learns to perform arithmetic tabula rasa . More seriously, it seems Deepmind and the AI community in general is having a Streetlight effect problem, i.e. looking for AI in what works now, rather than coming to terms with the hard challenges. This explains why there are so many papers on GANs. People are just doubling down on what works (where the streetlight is), rather than acknowledgin…

I wouldn't dismiss GANs so easily. Yann LeCun was singing odes to GANs - as the most interesting idea in the last decade. The interesting thing about GANs is that they don't use a predefined loss, but instead the discriminator acts as the loss function for the generator - thus, it is learning a loss fn instead of using human guesswork to create it. That's quite a powerful new idea. Applications of GANs include making simulated images look more real, which is essential for RL, generating 'artificial' training images for other tasks and using the discriminator as an image embedding generator or classifier.

Re: Mastering Chess and Shogi by Self-Play with General Reinforcement Learning

#142

Post by Isofarro »

SubiculumCode wrote: post #40
luckyt wrote: post #28

It doesn't seem to like the Sicilian Defense (1.e4 c5), which is the most popular opening by human players. I wonder if this will change opening theory?

That's stunning. I thought that was one of the strongest openings for black.

Not strongest opening, but because it's an asymmetric opening system, which introduces imbalance into the position, thus it tends to have less drawish tendencies than a symmetric opening system.

This creates the psychological effect of slightly turning the knob of "Black is playing for equality", to "Black is playing for counter-play".

Re: Mastering Chess and Shogi by Self-Play with General Reinforcement Learning

#143

Post by CydeWeys »

allenz wrote: post #64
gwern wrote: post #30

Earlier quoted context omitted.

> So in a way, the real triumph of the AlphaGo series was the TPU and GPU army. Eh. It's still searching many fewer positions than Stockfish is.

It searches fewer positions because it decides where to search using 4 TPUs, which are 180 teraflops each according to Google.

TPUs aren't "cheating" though, as they can be used for generalized machine learning models, and not just Go.

Computer graphics is still an impressive achievement even when done on a GPU instead of a CPU.

Re: Mastering Chess and Shogi by Self-Play with General Reinforcement Learning

#144

Post by EvgeniyZh »

soveran wrote: post #121

The ten sample games: Sample game 1 https://lichess.org/VMe0gfa2 Sample game 2 https://lichess.org/Zqwn4Gzk Sample game 3 https://lichess.org/G2fPHci8 Sample game 4 https://lichess.org/LLt8wyYp Sample game 5 https://lichess.org/3r6CXx3H Sample game 6 https://lichess.org/sbdyUYS4 Sample game 7 https://lichess.org/88vsAftE Sample game 8 https://lichess.org/1uvCwaeB Sample game 9 https://lichess.org/743quCXj Sample game…

So, 1. d4 for White, Berlin for Black. I got it

Re: Mastering Chess and Shogi by Self-Play with General Reinforcement Learning

#145

Post by CydeWeys »

grumpopotamus wrote: post #55
gwern wrote: post #10

This is an incredible demonstration that the AG Zero expert iteration method is a general method. If you go back to the discussions of AG Zero lo a month ago, there was a lot of skepticism that NNs would ever challenge Stockfish et al - they are just too good, too close to perfection, and chess not well suited for MCTS and NNs. Well, it turns out that AG Zero doesn't work as well in chess: it works better as it only…

>it only takes 4 hours of training to beat Stockfish In that time I figure they used the equivalent of about 1000 cpu-years. Imagine the things we'll be able to achieve as we can do more and more computation in less and less time.

> In that time I figure they used the equivalent of about 1000 cpu-years.

Are you using some kind of conversion factor from TPUs to CPUs? If so, what is it? And is it valid to do that?

You could convert the amount of time it took to render an hour's worth of gameplay from 1 GPU-hour to 50 CPU-days (or whatever), but is that really meaningful?

Re: Mastering Chess and Shogi by Self-Play with General Reinforcement Learning

#146

Post by allenz »

nl wrote: post #101
allenz wrote: post #64

Earlier quoted context omitted.

It searches fewer positions because it decides where to search using 4 TPUs, which are 180 teraflops each according to Google.

The number of positions evaluated is the number evaluated. Speed doesn’t change that. Speed probably made the initial self play training quicker though.

Compute absolutely matters. With tree search, there's a tradeoff between scoring cost and positions evaluated. AlphaZero can evaluate fewer positions because it uses a huge amount of compute to accurately score each position.

It's not just training. Training used 5,000 TPUs.

Re: Mastering Chess and Shogi by Self-Play with General Reinforcement Learning

#147

Post by CydeWeys »

cdancette wrote: post #105
grumpopotamus wrote: post #55

Earlier quoted context omitted.

>it only takes 4 hours of training to beat Stockfish In that time I figure they used the equivalent of about 1000 cpu-years. Imagine the things we'll be able to achieve as we can do more and more computation in less and less time.

But how many "cpu hours" of human work were used to design stockfish? You can't really compare that. Some scientist say the brain has a power of several petaflops, so if you use this, I guess the design of stockfish was way less efficient. You can't really compare things to cpu years, it doesn't make sense. Power consumption would be a better metric I think.

The best metric is total cost, including the cost of the hardware as well as the electricity. It might be worth prorating the hardware by the amount of time it spends on the task, too, assuming the hardware is general enough for many purposes (like TPUs are), vs say something like EFF's DES cracker which was not.

Re: Mastering Chess and Shogi by Self-Play with General Reinforcement Learning

#148

Post by EvgeniyZh »

1024core wrote: post #51
xianshou wrote: post #25

One impressive statistic from the paper: AlphaZero analyzes 80,000 chess positions per second, while Stockfish looks at 70,000,000. Seventy million, three orders of magnitude higher. Yet AG0 beats Stockfish half the time as White and never loses with either color. A stunning demonstration of generality indeed.

So ... what if you combined Stockfish and AG0, and let AG0 explore 70M positions instead of 80K? Would it improve even faster?

What if you combined a bus that gets you to work in 10 minutes and plane that gets you from Paris to Brazil, would it get you from Paris to Brazil in 10 minutes?

Re: Mastering Chess and Shogi by Self-Play with General Reinforcement Learning

#149

Post by hyperbovine »

Aissen wrote: post #106

Serious question: how does one evaluate the results reproducibility of this paper ? Maybe I'm missing some things but: - Are 1st gen TPUs even accessible ? You have to fill out a form to learn more about those second generation TPUs: https://cloud.google.com/tpu/ - I can't find the source code This does not look like a scientific paper, but a ( very impressive) tech demo.

Definitely a scientific paper. It’s obvious from they way they formatted it that they’re going to submit this to Nature or Science.

Re: Mastering Chess and Shogi by Self-Play with General Reinforcement Learning

#150

Post by thomasahle »

thom wrote: post #65

I can’t see any reference to whether Stockfish was configured with an endgame tablebase. It’d be interesting to see results then, as you’d expect AlphaZero’s superior evaluation to give it an advantage out of the opening, but later in the game Stockfish would have access to perfect evaluations. Obviously there’s nothing stopping you from plugging a tablebase into AlphaZero but that feels wrong.

It's not clear that it had an opening book either. In any case it's not specified which one.
Post reply on HN