Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Mastering Chess and Shogi by Self-Play with General Reinforcement Learning

arxiv.org

Post reply on HN
Print view View original

191–200 of 282 posts

Re: Mastering Chess and Shogi by Self-Play with General Reinforcement Learning

#191

Post by jmmcd »

2bitencryption wrote: post #8
ericand wrote: post #2

Certainly a significant achievement. Also, kind of interesting that the AlphaGo team spent a lot of energy to convince us Go is much harder than Chess, only to turn around and tell us that it is amazing that it can also win at Chess.

> only to turn around and tell us that it is amazing that it can also win at Chess. What they're demoing here is a single, general formula for mastering multiple games. Start with empty AG0, then teach it chess from scratch until it is the strongest player on the planet. Go back to an empty slate, with the same exactly "untrained" AG0, and now teach it Go, to the same result. No fine-tuning for the domain of the game…

> Go back to an empty slate, with the same exactly "untrained" AG0, and now teach it Go, to the same result. No fine-tuning for the domain of the game you are training -- it is general(ized).

Not quite -- different input features, which implies slightly different network architecture at least at the front.

Re: Mastering Chess and Shogi by Self-Play with General Reinforcement Learning

#192

Post by dragonwriter »

tantalor wrote: post #190
gwern wrote: post #10

This is an incredible demonstration that the AG Zero expert iteration method is a general method. If you go back to the discussions of AG Zero lo a month ago, there was a lot of skepticism that NNs would ever challenge Stockfish et al - they are just too good, too close to perfection, and chess not well suited for MCTS and NNs. Well, it turns out that AG Zero doesn't work as well in chess: it works better as it only…

> doesn't work as well in chess: it works better That's not how "as well" works.

Yes, it is; while the idiom standing on its own implicitly includes a leading “at least”, it is also idiomatic to use it in exactly the way used by the grandparent post, in an explicit contrast with better, where it comes with an implicit (or sometimes explicit) leading “merely” instead of “at least”.

Re: Mastering Chess and Shogi by Self-Play with General Reinforcement Learning

#193

Post by setr »

nl wrote: post #186
arcticfox wrote: post #172

Earlier quoted context omitted.

Can't those teraflops be applied to evaluating more positions instead of deciding which positions to evaluate? It seems that the metric should be compute time, not positions evaluated.

Presumably they both had equal clock time - that is a standard chess rule so it would be surprising to see it different.

Wall clock != cpu clock

I can do more in the same wall time with a faster cpu(s); I can afford inefficiencies that the opponent cannot, and accomplish just as much.

Re: Mastering Chess and Shogi by Self-Play with General Reinforcement Learning

#194

Post by nopinsight »

Cybiote wrote: post #174
Cybiote wrote: post #69

Earlier quoted context omitted.

Let's break this down and consider things carefully. To informed researchers, what is most surprising here is not that the AlphaGo Zero algorithm beat stockfish but that MCTS managed to outperform Alpha-beta search. I'll venture a hypothesis as to why this was. Informed skepticism would have discounted MCTS against alpha-beta search but wouldn't have put much stock into the idea that Neural Networks couldn't learn be…

Correction to the above: I stated Deepmind applied Neural Nets+MCTS and achieved ok results. I was actually misremembering two David Silver (Deepmind) papers as one. Smooth UCT modified UCT (popular brand of MCTS) to be able to handle imperfect information games. MCTS does not converge under imperfect information. Smooth UCT is strong at limit poker . Limit is much simpler than no-limit. Neural Fictitious Self Play b…

What do you think of Libratus which won quite convincingly against top players in no-limit Texas hold ‘em poker?

https://en.m.wikipedia.org/wiki/Libratus

Re: Mastering Chess and Shogi by Self-Play with General Reinforcement Learning

#195

Post by hmate9 »

sanxiyn wrote: post #182
hmate9 wrote: post #162

So AlphaGo Zero used 4 TPUs while AlphaZero used 1500. It’s not immediately obvious to me why there is this massive difference. Can anyone elaborate?

Both used 4 TPUs at playing time. At training time, AlphaGo Zero used unspecified amount of computing resource, AlphaZero used 5000 TPUs for self-play.

Ah, thanks for clearing that up! Makes sense.

Re: Mastering Chess and Shogi by Self-Play with General Reinforcement Learning

#196

Post by AlexCoventry »

elcapitan wrote: post #139
thanatropism wrote: post #132

Earlier quoted context omitted.

The keyword is "reinforcement learning".

I know the names of the general concepts, I was wondering if someone has concrete recommendations on where to start and which books/frameworks are sort of beginner-friendly.

Try Hands-On Machine Learning with Scikit-Learn and TensorFlow Concepts, Tools, and Techniques to Build Intelligent Systems for the fundamentals.

http://shop.oreilly.com/product/0636920052289.do

For reinforcement learning, I hear Barto and Sutton is very readable, but I haven't read it myself. You can just pick the concepts up by reading papers. The introduction in the Deep Q-Learning paper is not great, but it's how I first learned the concept.

http://ufal.mff.cuni.cz/~straka/courses/npfl114/2016/sutton-... https://www.cs.toronto.edu/~vmnih/docs/dqn.pdf

Re: Mastering Chess and Shogi by Self-Play with General Reinforcement Learning

#198

Post by Retric »

CydeWeys wrote: post #187
Retric wrote: post #178

Earlier quoted context omitted.

I think his point is if you devote X Flops to something then a fair comparison would be to also give X Flops to the competitor. The specifics of how an algorithm does not matter as much as the total resources used and outcome.

A more fair comparison would be to cap the hardware used at a certain cost. That's much more reflective of the real world. There are plenty of tasks that perhaps you could do more efficiently on a CPU for a given number of operations, e.g. maybe some graphics operations, but in practice it's completely irrelevant because a GPU gives so much more performance for the given cost. There's nothing special about an operati…

Only if you're buying hardware based on the algorithm used. Useful chess programs need to actually run on people's phones where performance on a cluster of ASIC's is mostly meaningless.

Re: Mastering Chess and Shogi by Self-Play with General Reinforcement Learning

#199

Post by T-hawk »

dragonwriter wrote: post #192
tantalor wrote: post #190

Earlier quoted context omitted.

> doesn't work as well in chess: it works better That's not how "as well" works.

Yes, it is; while the idiom standing on its own implicitly includes a leading “at least”, it is also idiomatic to use it in exactly the way used by the grandparent post, in an explicit contrast with better, where it comes with an implicit (or sometimes explicit) leading “merely” instead of “at least”.

It's unnecessary, though, and makes the point harder to read. "It works even better" would be a perfectly sufficient description. "It works not as well as but better" is an unnecessary rhetorical flourish.

Re: Mastering Chess and Shogi by Self-Play with General Reinforcement Learning

#200

Post by gwern »

allenz wrote: post #64
gwern wrote: post #30

Earlier quoted context omitted.

> So in a way, the real triumph of the AlphaGo series was the TPU and GPU army. Eh. It's still searching many fewer positions than Stockfish is.

It searches fewer positions because it decides where to search using 4 TPUs, which are 180 teraflops each according to Google.

But better evaluation gives you asymptotic speedups. You can give Stockfish several times its computation (which is already a lot, I mean, 64 threads, come on) and it doesn't make good use of it since it just runs into the search wall. If you gave Stockfish the equivalent in CPU power (and I'm not sure this is a fair hypothetical since part of the appeal of NNs is that they have such efficient hardware implementations, so it seems unfair to then grant a less efficient algorithm equivalent computing power by fiat), I'm not sure it would be restored to parity or superiority.
Post reply on HN