Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Alpha Go Zero: How and Why It Works

tim.hibal.org

Post reply on HN
Print view View original

81–90 of 116 posts

Re: Alpha Go Zero: How and Why It Works

#81

Post by eru »

shghs wrote: post #26

The main reason AlphaGo Zero learns so much faster than its predecessors is because it uses temporal-difference learning.[1] This effectively removes a huge amount of the value network's state space for the learning algorithm to search through, since it bakes in the assumption that a move's value ought to equal that of the best available move in the following board position, which is exactly what you'd expect for a g…

Of course, these techniques you have mentioned were known before and tried. The big thing in AlphaGo Zero is that they found a way to make them work and the resulting architecture even manages to looks simple.

(Eg AlphaGo using two different networks for policy and valuation was a big breakthrough back when it was young, because people couldn't make the single network one work so well.)

For temporal difference learning the article about TD-Gammon, a backgammon AI from the early 90s, is great: http://www.bkgm.com/articles/tesauro/tdl.html (It's linked from the Wikipedia article you referenced, too.)

Re: Alpha Go Zero: How and Why It Works

#83

Post by erikb »

I don't get what is new in the set of attributes that this article describes.

Monte Carlo was already used in 2005 in AIs playing on KGS. Gradient Descent is a basic algorithm is a basic algorithm that I saw in an AI class in ~2008 as well. I bet both are even a lot older and well known by all experts.

This is not what makes AlphaGo special or Zero successful. The curious thing about Zero is that usually with Gradient Descent you run a huge risk of running into a local maximum and then stop evolving because every evolution makes you not better than the current step.

So one question is actually how they used these same old algorithms so much more efficiently, and the second question is how did they overcame the local maximum problem. Additionally there may be other problems involved that experts know better than me.

But an explanation of basic algorthims can't be the answer.

Re: Alpha Go Zero: How and Why It Works

#84

Post by gwern »

erikb wrote: post #83

I don't get what is new in the set of attributes that this article describes. Monte Carlo was already used in 2005 in AIs playing on KGS. Gradient Descent is a basic algorithm is a basic algorithm that I saw in an AI class in ~2008 as well. I bet both are even a lot older and well known by all experts. This is not what makes AlphaGo special or Zero successful. The curious thing about Zero is that usually with Gradien…

> The curious thing about Zero is that usually with Gradient Descent you run a huge risk of running into a local maximum and then stop evolving because every evolution makes you not better than the current step.

No. The curious thing is that you can train a godawful huge NN with 40 layers via pure self-play with no checkpoints or baselines or pretraining or library of hard problems or any kind of stabilization mechanism, and it won't diverge but will learn incredibly rapidly and well and stably. As Silver says in the AmA, all their attempts at pure self-play ran into the usual divergence problems where the training explodes and engages in catastrophic forgetting, which is what the RL folklore predicts will happen if you try to do that. Local maximums are not the problem - the problem is the self-play can't even find a local maximum much less maintain it or improve it.

> So one question is actually how they used these same old algorithms so much more efficiently, and the second question is how did they overcame the local maximum problem.

Er, this is exactly what OP is all about: the Monte Carlo tree search supervision. That's how they used them.

Re: Alpha Go Zero: How and Why It Works

#85

Post by praptak »

shghs wrote: post #26

The main reason AlphaGo Zero learns so much faster than its predecessors is because it uses temporal-difference learning.[1] This effectively removes a huge amount of the value network's state space for the learning algorithm to search through, since it bakes in the assumption that a move's value ought to equal that of the best available move in the following board position, which is exactly what you'd expect for a g…

The seminal article on TDRL: Sutton, R. (1988). Learning to predict by the methods of temporal differences. Machine learning, 3(1):9–34, PDF:http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.132...

Re: Alpha Go Zero: How and Why It Works

#86

Post by alkonaut »

falcor84 wrote: post #82

As someone who has played many a game of Tic-Tac-Toe, I found the numerical examples really hard to follow. s(0,5) is obviously the winning move for the X player, but for some reason all examples seem to favor s(0,1).

Winning move how? With which board?

with

    O__
    _X_
    O_X
Then X can't win in the next move and must choose (0,1) between the two O's the left column in order to not lose.

Re: Alpha Go Zero: How and Why It Works

#88

Post by nojvek »

nemo1618 wrote: post #57

Would be really cool to see a generic framework for this, where you can plug in the rules of your discrete-deterministic-game-with-perfect-information and get a superhuman bot. Does something like this already exist?

If anyone's interested who understands alpha zero in depth. I would love to start a github project to implement a super not that players checkers, tic tac toe, chess, go etc and all related games on the browser via an openai like interface.

Re: Alpha Go Zero: How and Why It Works

#89

Post by nojvek »

naveen99 wrote: post #79
zeristor wrote: post #70

Are there any plans to do this for Chess? I imagine that this is an iteration of the Alpha Go engine, people working on this are very current with Alpha Go. If Chess is similar, then wouldn't DeepMind be able to bootstrap game knowledge. Perhaps this isn't a big goal, but Chess is Chess after all.

There was a project for chess, called giraffe https://bitbucket.org/waterreaction/giraffe whose author shut it down after joining google deep mind: http://www.talkchess.com/forum/viewtopic.php?t=59003 He thinks it's only a matter of time till machine learning beats hand crafted systems like stockfish even in chess. http://arxiv.org/abs/1509.01549

Does alphazero beat stock fish purely by self play. That would be huge. Stock fish is the result of so many hand crafted optimizations over a large game play dataset.

Re: Alpha Go Zero: How and Why It Works

#90

Post by nojvek »

unpseudo wrote: post #62

They are two things a human brain does when playing chess or go: evaluating a position and mentally playing some positions (by doing a search tree). The AlphaGo neural network is able to do the first part (evaluating positions) but the search tree is still a hand crafted algorithm. Do they have plans to work on a version with a pure neural network? (i.e. a version which would be able to learn how to do a search tree.…

Who is to say that the human brain doesn't have a different cell type to do this kind of search. The neocortex is shaped in a different way than other neural cells so I imagine search and evaluation are different architectures.

But you're right. A NN way to do Montecarlo search in GPUs would make things even simpler.

Post reply on HN