Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Alpha Go Zero: How and Why It Works

tim.hibal.org

Post reply on HN
Print view View original

21–30 of 116 posts

Re: Alpha Go Zero: How and Why It Works

#21

Post by Barrin92 »

danielvf wrote: post #16
yters wrote: post #9

Are there adversarial examples for Alpha Go Zero?

What do you mean? AlphaGo Zero has played itself, and they have published 20 of those games.

the OP is referring to the use of a GAN, which is a different type of setup.

GAN's were not used for alphaGo, as the article points out Deepmind uses reinforcement learning and MCTS.

Re: Alpha Go Zero: How and Why It Works

#22

Post by gwern »

QML wrote: post #6

What makes this different from a minimax algorithm with alpha-beta pruning?

Aside from MCTS being a different tree search method, there is no 'closing of the loop'. In regular MCTS, it is far from unheard of to do the random playouts with instead some 'heavier' heuristic to make the playouts a little better estimators of the node value, but the heavy playouts do not do any kind of learning, the heuristic you start with is what you end with; what makes this analogous to policy iteration (hence the names for the Zero algorithm of 'tree iteration' or 'expert iteration') is that the refined estimates from the multiple heavy playouts are then used to improve the heavy playout heuristic (ie. a NN which can be optimized via backpropagation in a supervised learning of board position -> value). Then in a self-play setting, the MCTS continually refines its heavy heuristic (the NN) until it's so good that the NN+MCTS is superhuman. Then at play time you can drop the MCTS entirely and just use the heavy heuristic to do a very simple tree search to choose a move (which I think might actually be a minimax with a fixed depth but I forget).

Re: Alpha Go Zero: How and Why It Works

#23

Post by javajosh »

ben_w wrote: post #8
javajosh wrote: post #3

"On a long enough timeline, everything is a discrete game." (With apologies to _Fight Club_) Personally, I look forward to the day when the software I own works for me to the extent of optimizing the decisions I make during the day, even many mundane ones. Properly executed, such a system could make a big difference in my quality of life. I believe that a big piece that is missing is a solid life-model, a life-repres…

If it is optimising all your decisions, even the mundane ones, is it really your life any more? If it plugged into your brainstem and took over, by definition nobody else would notice the difference (you always did what it said anyway, or you’d be suboptimal), so why not just flood the 20 watt carbon-based neural network with bliss drugs and let the silicon-based neural network be the person instead?

I think sometimes it's hard for us to zoom out and solve bigger problems that we might not see. Like, if I don't enjoy the chore I'm doing, I can make it easier, or I can eliminate it. (Paying a bill in person is a good example). Or, maybe there is an object that is generating a lot of work for me, like a house, and there is an alternative object (an apartment). Heck, sometimes it would be nice to have a record of all the things I enjoy and just have an AI path-find to maximize revisiting those things.

Re: Alpha Go Zero: How and Why It Works

#24

Post by ImSkeptical »

badminton1 wrote: post #5

Go has been studied for hundreds of years. In many cases, by people who study the game since their childhood and work on it as a full-time occupation. The consequence of Alpha Go Zero is that it can, in a matter of days, disregard and surpass all human knowledge about the game. Maximizing a score margin has been equated for a long time with maximizing your probability of winning. Alpha Go doesn't play like that... it…

I think people have always understood the difference between maximising probability of victory versus score. Even in amateur games you'll get the feeling "I could fight here, and it would be complicated, but I might get a huge advantage, or I could just play safe and keep my advantage."

Re: Alpha Go Zero: How and Why It Works

#26

Post by shghs »

The main reason AlphaGo Zero learns so much faster than its predecessors is because it uses temporal-difference learning.[1] This effectively removes a huge amount of the value network's state space for the learning algorithm to search through, since it bakes in the assumption that a move's value ought to equal that of the best available move in the following board position, which is exactly what you'd expect for a game like Go.

A secondary reason for AlphaGo Zero's performance is that it combines both value and policy networks into a single network, since it's redundant to have two networks for move selection.

These are the two biggest distinguishing characteristics of AlphaGo Zero compared to previous AlphaGos, and the OP doesn't discuss either of them.

[1] https://en.wikipedia.org/wiki/Temporal_difference_learning

Re: Alpha Go Zero: How and Why It Works

#29

Post by Choco31415 »

jacobkg wrote: post #27

Can this technique be used to write a strong chess engine?

Yes. As described, all that’s needed is a way to imagine all possible moves from a game state, and check if a game state corresponds with win/tie/loss. That is possible in Tic-Tac-Toe, Chess, and Go.

Re: Alpha Go Zero: How and Why It Works

#30

Post by jeffshek »

javajosh wrote: post #3

"On a long enough timeline, everything is a discrete game." (With apologies to _Fight Club_) Personally, I look forward to the day when the software I own works for me to the extent of optimizing the decisions I make during the day, even many mundane ones. Properly executed, such a system could make a big difference in my quality of life. I believe that a big piece that is missing is a solid life-model, a life-repres…

Site : https://betterself.io/ GitHub : https://github.com/jeffshek/betterself

I do this similarly but with supplements and medication. I track a lot of my supplements that boost productivity and sleep and disregard anything that's negative. I've been continuing this over the course of a year.

I haven't added anything like a RNN to it for recommendations though. Some fears about missing up the gradient descent component scare me :)

Post reply on HN