Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Mastering Chess and Shogi by Self-Play with General Reinforcement Learning

arxiv.org

Post reply on HN
Print view View original

171–180 of 282 posts

Re: Mastering Chess and Shogi by Self-Play with General Reinforcement Learning

#171

Post by thom »

bhouston wrote: post #170
thom wrote: post #97

Earlier quoted context omitted.

It would be interesting to see if there were some way to extract a couple of new heuristics from AlphaZero that could be implemented fast enough to incorporate in Stockfish's evaluator though. I suppose this is the age old problem of black-box models: _why_ does it think this?

I think that it is almost always possible to extract optimized models from nn and implement them faster. I wonder if this can be generalized. Nn to optimized fix algo for Max speed? This has to already exist as it is very obvious.

I dunno, seems like Google would just do this instead of keep around the pesky neural net at runtime. There's an _awful_ lot of computation going on inside, and it's necessarily hugely interconnected. I'd be impressed if someone had already done it, but it seems a great avenue of research if not. I suppose it goes hand in hand with models for which you can actually _explain_ their results, which certainly is an active area of research.

Re: Mastering Chess and Shogi by Self-Play with General Reinforcement Learning

#172

Post by arcticfox »

nl wrote: post #101
allenz wrote: post #64

Earlier quoted context omitted.

It searches fewer positions because it decides where to search using 4 TPUs, which are 180 teraflops each according to Google.

The number of positions evaluated is the number evaluated. Speed doesn’t change that. Speed probably made the initial self play training quicker though.

Can't those teraflops be applied to evaluating more positions instead of deciding which positions to evaluate?

It seems that the metric should be compute time, not positions evaluated.

Re: Mastering Chess and Shogi by Self-Play with General Reinforcement Learning

#173

Post by thomasahle »

thom wrote: post #80
zwischenzug wrote: post #79

Earlier quoted context omitted.

Now I'm really intrigued. SF plays really odd moves when left to its own devices for a time. As does this AI. So maybe chess looks really weird with play significantly better than the best humans. It's actually really disturbing.

I think being able to play tactically perfect chess over 20 or so moves will often look weird to human strategic sensibilities. The computer sees every tiny exception to the patterns and heuristics you've incorporated into your gut feel about positions. In a way these moves are right just because they're right, and that's what's jarring - there's no _principle_ behind them that can be learned and generalised, which i…

Except AlphaZero doesn't evaluate nearly as many moves as Stockfish (80Knps vs 70Mnps), so in a sense, it has exactly generalized a principle (or likely a whole lot of principles) that allows it to estimate positions much better than Stockfish.

Of course you are right about perfect play, but the human-like aspect is part of what is exciting about these new Alpha engines.

Re: Mastering Chess and Shogi by Self-Play with General Reinforcement Learning

#174

Post by Cybiote »

Cybiote wrote: post #69
gwern wrote: post #10

This is an incredible demonstration that the AG Zero expert iteration method is a general method. If you go back to the discussions of AG Zero lo a month ago, there was a lot of skepticism that NNs would ever challenge Stockfish et al - they are just too good, too close to perfection, and chess not well suited for MCTS and NNs. Well, it turns out that AG Zero doesn't work as well in chess: it works better as it only…

Let's break this down and consider things carefully. To informed researchers, what is most surprising here is not that the AlphaGo Zero algorithm beat stockfish but that MCTS managed to outperform Alpha-beta search. I'll venture a hypothesis as to why this was. Informed skepticism would have discounted MCTS against alpha-beta search but wouldn't have put much stock into the idea that Neural Networks couldn't learn be…

Correction to the above: I stated Deepmind applied Neural Nets+MCTS and achieved ok results. I was actually misremembering two David Silver (Deepmind) papers as one. Smooth UCT modified UCT (popular brand of MCTS) to be able to handle imperfect information games. MCTS does not converge under imperfect information. Smooth UCT is strong at limit poker. Limit is much simpler than no-limit.

Neural Fictitious Self Play based on fictitious play (invented 1950s), is an approach to reinforcement learning using neural nets for function approximation. Typical RL methods like DQN are highly exploitable. Against strong programs, NFSP did okay, with a win rate of -50 mbb/h against the best bot it played against.

Looking not just at Deepmind, there's Deepstack. It's similar to AlphaGo OG, combining CFR+Neural nets. Deepstack did not win convincingly against humans at 2 player no limit hold em.

The general point I'm trying to make here is that Chess and Go are closer to checkers than to poker, which is itself a constrained game with known rules. I mention all this and this Deepmind paper: https://arxiv.org/pdf/1711.00832.pdf, to provide a sense of scale to those talking about smoke and fire alarms.

Re: Mastering Chess and Shogi by Self-Play with General Reinforcement Learning

#175

Post by naveen99 »

visarga wrote: post #11
naveen99 wrote: post #7

Very happy to see this result. It's like a moral victory for humans, as alphago is more human like (discounting montecarlo search) than stockfish. Maybe deep learning will give us the next Euler, Newton, or Einstein.

Shogi, chess and Go are "perfect information games", meaning you can see the whole game state. It's a whole different thing to be able to solve games where you don't see everything (based on uncertainty).

Math can be perfect information too if you just start with axioms. Even when starting with conjectures, the rules are transparent for manipulating statements.

Re: Mastering Chess and Shogi by Self-Play with General Reinforcement Learning

#178

Post by Retric »

CydeWeys wrote: post #143
allenz wrote: post #64

Earlier quoted context omitted.

It searches fewer positions because it decides where to search using 4 TPUs, which are 180 teraflops each according to Google.

TPUs aren't "cheating" though, as they can be used for generalized machine learning models, and not just Go. Computer graphics is still an impressive achievement even when done on a GPU instead of a CPU.

I think his point is if you devote X Flops to something then a fair comparison would be to also give X Flops to the competitor. The specifics of how an algorithm does not matter as much as the total resources used and outcome.

Re: Mastering Chess and Shogi by Self-Play with General Reinforcement Learning

#179

Post by sndean »

hyperbovine wrote: post #149
Aissen wrote: post #106

Serious question: how does one evaluate the results reproducibility of this paper ? Maybe I'm missing some things but: - Are 1st gen TPUs even accessible ? You have to fill out a form to learn more about those second generation TPUs: https://cloud.google.com/tpu/ - I can't find the source code This does not look like a scientific paper, but a ( very impressive) tech demo.

Definitely a scientific paper. It’s obvious from they way they formatted it that they’re going to submit this to Nature or Science.

This then may suggest that there’ll be this detail-light manuscript in the journal and a 50-100 page supplemental document available to download, with all the details to reproduce (hopefully)

Re: Mastering Chess and Shogi by Self-Play with General Reinforcement Learning

#180

Post by Retric »

deepnotderp wrote: post #36
gwern wrote: post #30

Earlier quoted context omitted.

> So in a way, the real triumph of the AlphaGo series was the TPU and GPU army. Eh. It's still searching many fewer positions than Stockfish is.

Right right, but my comparison was between giraffe and AlphaGo , not neural networks and Stockfish.

By starving the competitor of computing power, if you compare A and B you can't give A 10+x the compute power and assume a fair comparison. What's interesting is a demonstration that enough compute power let's NN reach beyond human level play. Though, I don't think that was ever really in doubt.
Post reply on HN