Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Mastering Chess and Shogi by Self-Play with General Reinforcement Learning

arxiv.org

Post reply on HN
Print view View original

41–50 of 282 posts

Re: Mastering Chess and Shogi by Self-Play with General Reinforcement Learning

#41

Post by throw_away_777 »

tlb wrote: post #37
Houshalter wrote: post #15

Earlier quoted context omitted.

It's not. It learns entirely through self play and never learns from playing it's opponent. Diminishing returns isn't unusual and happens in every domain. These AIs are probably playing close to the limit of what is possible, just not quite there yet.

Are there popular games where the best human players are not near the limit of what is possible? Obviously you can construct one to be hard for humans (large 3SAT problems, or even big arithmetic problems), but I wonder if there is one that people enjoy.

Humans are nowhere near the limit of what is possible in chess, as evidenced by how much better computers are at the game.

Re: Mastering Chess and Shogi by Self-Play with General Reinforcement Learning

#43

Post by gamegoblin »

badminton1 wrote: post #5

If you have seen the Stockfish project you will see many hardcoded weights in the configuration, found through experimentation. All these adjustments took probably years to achieve... and now Alpha Go Zero just self-learns everything and surpasses it. Would be good to see Deepmind's solution play Arimaa and Stratego, and see what kind of strategy it comes up with. Or weird variations of Go. Eventually this tech will…

I don't think this technique immediately applies to Stratego because it's not a perfect information game.

I suspect it would exceed the state of the art in Arimaa, since Arimaa is specifically designed to have a high branching factor (17281 -- compared to 35 for chess), and this technique was designed to work well in high-branching factor games (since Go is a high-branching factor game, though much lower than Arimaa).

Re: Mastering Chess and Shogi by Self-Play with General Reinforcement Learning

#44

Post by SubiculumCode »

SubiculumCode wrote: post #40
luckyt wrote: post #28

It doesn't seem to like the Sicilian Defense (1.e4 c5), which is the most popular opening by human players. I wonder if this will change opening theory?

That's stunning. I thought that was one of the strongest openings for black.

It seemed to play a lot of English Opening...that also seems strange to me.

Re: Mastering Chess and Shogi by Self-Play with General Reinforcement Learning

#45

Post by tboerstad »

Stockfish plays like an ambitious amateur in the first game, giving away a piece for two pawns on move 13.

Perhaps this move was justified though, as later in the same game Stockfish gets a position which is at worst drawn, likely winning. Moves later however, around move 40, Stockfish gets its own knight trapped and the game is over.

This is not the kind of chess we normally see from Stockfish.

Re: Mastering Chess and Shogi by Self-Play with General Reinforcement Learning

#46

Post by Houshalter »

chillee wrote: post #35
gwern wrote: post #31

Earlier quoted context omitted.

No. DM only occasionally releases software. Expert iteration is simple enough that someone can code it up on their own and there's already a few clones, so if anyone cares to train their own, it's doable, although it may take a while.

"a while" is a bit of an understatement. Leela zero (the main alphago zero replication project) is a crowd sourced computation effort that's going to take a fairly long time to get anywhere. And from this paper: > "Training proceeded for 700,000 steps (mini-batches of size 4,096) starting from randomly initialised parameters, using 5,000 first-generation TPUs (15) to generate self-play games and 64 second-generation…

You don't have to start from zero though. It's cool that it works with google scale resources. But it seems like it would be faster to initialize with a neural net first trained to mimic the moves of an existing chess or Go AI. And then improve it from there.

>"Why is the net wired randomly?", asked Minsky. "I do not want it to have any preconceptions of how to play", Sussman said. Minsky then shut his eyes. "Why do you close your eyes?", Sussman asked his teacher. "So that the room will be empty." At that moment, Sussman was enlightened.

Re: Mastering Chess and Shogi by Self-Play with General Reinforcement Learning

#47

Post by backpropaganda »

gwern wrote: post #10

This is an incredible demonstration that the AG Zero expert iteration method is a general method. If you go back to the discussions of AG Zero lo a month ago, there was a lot of skepticism that NNs would ever challenge Stockfish et al - they are just too good, too close to perfection, and chess not well suited for MCTS and NNs. Well, it turns out that AG Zero doesn't work as well in chess: it works better as it only…

While I'm glad to see you're excited about this, take note that this is still an approach which requires that an exact model is known, the state is fully visible, and the reward is perfectly define-able and known. Progress in this setup isn't necessarily correlated with the kind of AI for which we'd need a fire alarm.

Re: Mastering Chess and Shogi by Self-Play with General Reinforcement Learning

#48

Post by chillee »

Houshalter wrote: post #46
chillee wrote: post #35

Earlier quoted context omitted.

"a while" is a bit of an understatement. Leela zero (the main alphago zero replication project) is a crowd sourced computation effort that's going to take a fairly long time to get anywhere. And from this paper: > "Training proceeded for 700,000 steps (mini-batches of size 4,096) starting from randomly initialised parameters, using 5,000 first-generation TPUs (15) to generate self-play games and 64 second-generation…

You don't have to start from zero though. It's cool that it works with google scale resources. But it seems like it would be faster to initialize with a neural net first trained to mimic the moves of an existing chess or Go AI. And then improve it from there. >"Why is the net wired randomly?", asked Minsky. "I do not want it to have any preconceptions of how to play", Sussman said. Minsky then shut his eyes. "Why do…

I don't think it's definitely true that will work well. AlphaZero did significantly better than the original versions of AlphaGo (which did learn from existing human games). However, even training those nets will still take a fairly intensive amount of computational resources.

As for that koan, I'm not convinced it's very applicable here. My interpretation of the koan is that the entire setup (training process, structure, etc.) all encode domain knowledge. In this case, I think AlphaZero's domain knowledge is transferable enough that I don't think it's relevant.

Re: Mastering Chess and Shogi by Self-Play with General Reinforcement Learning

#49

Post by roenxi »

visarga wrote: post #11
naveen99 wrote: post #7

Very happy to see this result. It's like a moral victory for humans, as alphago is more human like (discounting montecarlo search) than stockfish. Maybe deep learning will give us the next Euler, Newton, or Einstein.

Shogi, chess and Go are "perfect information games", meaning you can see the whole game state. It's a whole different thing to be able to solve games where you don't see everything (based on uncertainty).

Is it really though?

A big class of imperfect information games can be modeled by having a record of everything the agent has seen so far. Then it has exactly the same, if not more, information available than a human player in the same position. We know that with equal information AIs can make better decisions than humans (see also, AlphaGo :] ) so at that point the AI could reasonably be expected to achieve superhuman performance.

The "imperfect information games are harder for AI" crowd are going to be surprised by just how badly humans deal with imperfect information. AIs have a much better memory than humans do, and much more potential to use actual probability which humans are truly shocking at utilising (although neural networks don't seem to utilise this edge; so far).

Re: Mastering Chess and Shogi by Self-Play with General Reinforcement Learning

#50

Post by nopinsight »

maxander wrote: post #24
magoghm wrote: post #9

"We also analysed the relative performance of AlphaZero’s MCTS search compared to the state-of-the-art alpha-beta search engines used by Stockfish and Elmo. AlphaZero searches just 80 thousand positions per second in chess and 40 thousand in shogi, compared to 70 million for Stockfish and 35 million for Elmo. AlphaZero compensates for the lower number of evaluations by using its deep neural network to focus much more…

Meanwhile a human player considers But unsettlingly few, nonetheless.

Humans are also much weaker than AlphaZero in these three games. The difference in the numbers of positions searched might be responsible for a substantial part of that.
Post reply on HN