Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Mastering Chess and Shogi by Self-Play with General Reinforcement Learning

arxiv.org

Post reply on HN
Print view View original

91–100 of 282 posts

Re: Mastering Chess and Shogi by Self-Play with General Reinforcement Learning

#91

Post by nandemo »

Houshalter wrote: post #46
chillee wrote: post #35

Earlier quoted context omitted.

"a while" is a bit of an understatement. Leela zero (the main alphago zero replication project) is a crowd sourced computation effort that's going to take a fairly long time to get anywhere. And from this paper: > "Training proceeded for 700,000 steps (mini-batches of size 4,096) starting from randomly initialised parameters, using 5,000 first-generation TPUs (15) to generate self-play games and 64 second-generation…

You don't have to start from zero though. It's cool that it works with google scale resources. But it seems like it would be faster to initialize with a neural net first trained to mimic the moves of an existing chess or Go AI. And then improve it from there. >"Why is the net wired randomly?", asked Minsky. "I do not want it to have any preconceptions of how to play", Sussman said. Minsky then shut his eyes. "Why do…

I'm pretty sure starting from zero is the point of the Leela-Zero. If they started from Stockfish, it wouldn't be a replication of AlphaZero.

Re: Mastering Chess and Shogi by Self-Play with General Reinforcement Learning

#92

Post by [deleted] »

thom wrote: post #80
zwischenzug wrote: post #79

Earlier quoted context omitted.

Now I'm really intrigued. SF plays really odd moves when left to its own devices for a time. As does this AI. So maybe chess looks really weird with play significantly better than the best humans. It's actually really disturbing.

I think being able to play tactically perfect chess over 20 or so moves will often look weird to human strategic sensibilities. The computer sees every tiny exception to the patterns and heuristics you've incorporated into your gut feel about positions. In a way these moves are right just because they're right, and that's what's jarring - there's no _principle_ behind them that can be learned and generalised, which i…

[deleted]

Re: Mastering Chess and Shogi by Self-Play with General Reinforcement Learning

#93

Post by gcp »

Houshalter wrote: post #46
chillee wrote: post #35

Earlier quoted context omitted.

"a while" is a bit of an understatement. Leela zero (the main alphago zero replication project) is a crowd sourced computation effort that's going to take a fairly long time to get anywhere. And from this paper: > "Training proceeded for 700,000 steps (mini-batches of size 4,096) starting from randomly initialised parameters, using 5,000 first-generation TPUs (15) to generate self-play games and 64 second-generation…

You don't have to start from zero though. It's cool that it works with google scale resources. But it seems like it would be faster to initialize with a neural net first trained to mimic the moves of an existing chess or Go AI. And then improve it from there. >"Why is the net wired randomly?", asked Minsky. "I do not want it to have any preconceptions of how to play", Sussman said. Minsky then shut his eyes. "Why do…

The problem is that it isn't entirely clear whether this produces equal quality results. You might end up on a lower optimization plateau.

Re: Mastering Chess and Shogi by Self-Play with General Reinforcement Learning

#94

Post by gamegoblin »

asdfologist wrote: post #39

While this sounds impressive, I'll believe it when AlphaZero wins TCEC.

Back when AlphaGo was playing Lee Sedol I was thinking about a chess playing version in TCEC.

The interesting thing is TCEC assumes a bit about the structure of the chess program. That is, the TCEC win-adjudication rule says that if both programs agree that one program is 6.5 pawns ahead for 8 turns in a row, they judge that program to be the winner.

But programs like Alpha don't have an evaluation function that operates in conventional units (like centipawns).

Re: Mastering Chess and Shogi by Self-Play with General Reinforcement Learning

#95

Post by nandemo »

Invictus0 wrote: post #4

I'm not sure it's really fair to compare Stockfish to AlphaZero; AlphaZero used 24h of 5000 TPUs in compute time, and still needed 4 TPUs in real play, while Stockfish ran on just 64 threads and 1GB RAM. Nonetheless, still an impressive achievement.

Wait, how's the 24h x 5000 TPUs relevant? That is training time, and that training corresponds to years and years of hardcoding evaluations in Stockfish, not to compute time during the match.

Re: Mastering Chess and Shogi by Self-Play with General Reinforcement Learning

#96

Post by thom »

zwischenzug wrote: post #72
zwischenzug wrote: post #61

I smell a rat. The paper says: 'AlphaZero achieved within 24 hours a superhuman level of play in the games of chess and shogi' In the first game Stockfish's, 9. Qe1 is one of the strangest moves I've ever seen, which would never be considered by a human, let alone a superhuman. 11. Kh1 also makes little sense, but is not as bad. My Stockfish sees it as losing 0.2 pawns, which makes it highly suspect in such a positio…

If I leave Stockfish to study for longer then Qe1 comes up in the analysis. Which makes me wonder whether SF gets weaker in some positions the more it's left to think.

There's definitely nothing fishy going on, although it'd be nice to see a fully loaded Stockfish on its full complement of 512 cores and a proper endgame tablebase to really slog it out with AlphaZero.

Re: Mastering Chess and Shogi by Self-Play with General Reinforcement Learning

#97

Post by thom »

thom wrote: post #59
1024core wrote: post #51

Earlier quoted context omitted.

So ... what if you combined Stockfish and AG0, and let AG0 explore 70M positions instead of 80K? Would it improve even faster?

The issue is you can’t evaluate positions that fast in AlphaZero (currently).

It would be interesting to see if there were some way to extract a couple of new heuristics from AlphaZero that could be implemented fast enough to incorporate in Stockfish's evaluator though. I suppose this is the age old problem of black-box models: _why_ does it think this?

Re: Mastering Chess and Shogi by Self-Play with General Reinforcement Learning

#98

Post by thom »

thom wrote: post #96
zwischenzug wrote: post #72

Earlier quoted context omitted.

If I leave Stockfish to study for longer then Qe1 comes up in the analysis. Which makes me wonder whether SF gets weaker in some positions the more it's left to think.

There's definitely nothing fishy going on, although it'd be nice to see a fully loaded Stockfish on its full complement of 512 cores and a proper endgame tablebase to really slog it out with AlphaZero.

Back to zero at 41 ply... I shan't give a running commentary anymore.

Re: Mastering Chess and Shogi by Self-Play with General Reinforcement Learning

#99

Post by [deleted] »

deepnotderp wrote: post #27
gwern wrote: post #10

This is an incredible demonstration that the AG Zero expert iteration method is a general method. If you go back to the discussions of AG Zero lo a month ago, there was a lot of skepticism that NNs would ever challenge Stockfish et al - they are just too good, too close to perfection, and chess not well suited for MCTS and NNs. Well, it turns out that AG Zero doesn't work as well in chess: it works better as it only…

See the thing is though, Giraffe's evaluation actually was better than Stockfish's evaluation function, but it took much longer, and thus wasn't able to search as deep as Stockfish et al. So in a way, the real triumph of the AlphaGo series was the TPU and GPU army.

[deleted]

Re: Mastering Chess and Shogi by Self-Play with General Reinforcement Learning

#100

Post by Recursing »

Invictus0 wrote: post #4

I'm not sure it's really fair to compare Stockfish to AlphaZero; AlphaZero used 24h of 5000 TPUs in compute time, and still needed 4 TPUs in real play, while Stockfish ran on just 64 threads and 1GB RAM. Nonetheless, still an impressive achievement.

Only 1GB RAM? Really?
Post reply on HN