Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Alpha Go Zero: How and Why It Works

tim.hibal.org

Post reply on HN
Print view View original

101–110 of 116 posts

Re: Alpha Go Zero: How and Why It Works

#101

Post by psb217 »

gwern wrote: post #84
erikb wrote: post #83

I don't get what is new in the set of attributes that this article describes. Monte Carlo was already used in 2005 in AIs playing on KGS. Gradient Descent is a basic algorithm is a basic algorithm that I saw in an AI class in ~2008 as well. I bet both are even a lot older and well known by all experts. This is not what makes AlphaGo special or Zero successful. The curious thing about Zero is that usually with Gradien…

> The curious thing about Zero is that usually with Gradient Descent you run a huge risk of running into a local maximum and then stop evolving because every evolution makes you not better than the current step. No. The curious thing is that you can train a godawful huge NN with 40 layers via pure self-play with no checkpoints or baselines or pretraining or library of hard problems or any kind of stabilization mechan…

If you read the AGZ paper closely, they actually use checkpoints during training. Specifically, during training they only perform updates to the "stable" set of parameters when the current "learning" set of parameters produces a policy which beats the stable set at least 55% of the time. The current stable parameters are what they use for generating the self-play data which they use to update the current "learning" parameters. I believe this is only mentioned in the supplementary material...

Re: Alpha Go Zero: How and Why It Works

#102

Post by gugagore »

thomasahle wrote: post #80
pmarreck wrote: post #77

> It is interesting to see how quickly the field of AI is progressing. Those who claim we will be able to see the robot overlords coming in time should take heed - these AI's will only be human-level for a brief instant before blasting past us into superhuman territories, never to look back. This final paragraph is just editorializing. A computer will never care about anything (including games like Go and domination…

What do you mean with inverting a matrix? In the usual definition it is well studied and understood https://en.m.wikipedia.org/wiki/Invertible_matrix#Methods_of...

They might be talking about an interpretation of gradient descent that this article provides: http://blog.mrtz.org/2013/09/07/the-zen-of-gradient-descent....

Re: Alpha Go Zero: How and Why It Works

#103

Post by nvarsj »

abhgh wrote: post #55
dmix wrote: post #39

Earlier quoted context omitted.

Interesting, a TD algorithm, developed by a Canadian AI researcher now working with Deepmind in the early 1990s, was previously used to beat expert players at Backgammon and advanced human understanding of the game: > TD-Lambda is a learning algorithm invented by Richard S. Sutton based on earlier work on temporal difference learning by Arthur Samuel. This algorithm was famously applied by Gerald Tesauro to create TD…

TD-Gammon was taught to us as a part of a classroom course on Reinforcement Learning (RL) in 2007 . ML was known to a small set of people back then, there weren't many jobs in the area (this is in India), and even to many in this set, RL was either not known or not well known. It's interesting to see RL surge in popularity. In fact just a couple of weeks back, I was talking to the professor who taught us that course,…

Yes, me too. TD was also pretty much considered useless, until it was used for backgammon. And like many things in the AI/ML world, no one really knew exactly why it worked so well.

Backgammon is also interesting in that there is a non-determinstic element - the dice roll on every turn. This is where TD seems to shine.

Re: Alpha Go Zero: How and Why It Works

#104

Post by cosminro »

gwern wrote: post #32
shghs wrote: post #26

The main reason AlphaGo Zero learns so much faster than its predecessors is because it uses temporal-difference learning.[1] This effectively removes a huge amount of the value network's state space for the learning algorithm to search through, since it bakes in the assumption that a move's value ought to equal that of the best available move in the following board position, which is exactly what you'd expect for a g…

> These are the two biggest distinguishing characteristics of AlphaGo Zero compared to previous AlphaGos, and the OP doesn't discuss either of them. David Silver disagrees. The most critical distinguishing characteristic is the expert/tree iteration which makes stable self-play possible at all.

A lot of people reading the paper miss this. I guess it's not emphasized enough.

In the first paper, the selfplay trained policy is about 1500 in elo rating, while darkforest2 a supervised trained policy from Facebook is around the same, if not better. So selfplay wasn't of much use the first time around. While in the AlphaZero paper the selfplay trained policy has about 3000 elo rating.

Re: Alpha Go Zero: How and Why It Works

#105

Post by gwern »

psb217 wrote: post #101
gwern wrote: post #84

Earlier quoted context omitted.

> The curious thing about Zero is that usually with Gradient Descent you run a huge risk of running into a local maximum and then stop evolving because every evolution makes you not better than the current step. No. The curious thing is that you can train a godawful huge NN with 40 layers via pure self-play with no checkpoints or baselines or pretraining or library of hard problems or any kind of stabilization mechan…

If you read the AGZ paper closely, they actually use checkpoints during training. Specifically, during training they only perform updates to the "stable" set of parameters when the current "learning" set of parameters produces a policy which beats the stable set at least 55% of the time. The current stable parameters are what they use for generating the self-play data which they use to update the current "learning" p…

I did and I would point out that while they use checkpoints, the training curves indicate this is not necessary, and what I meant is that they do not use the usual self-play (and evolutionary) mechanism of checkpoints from throughout the training history which is necessary to combat catastrophic forgetting (and which apparently wasn't enough to stabilize Zero on its own as it is the single most obvious thing to do but Silver notes all the pre-Zero self-plays diverged until they finally came up with that of MCTS supervision). The checkpoint mechanism there appears no more necessary than checkpoints in training any NN - it's critical to avoid a random error or bug wasting weeks of time but does not affect the training dynamics in any important way.

Re: Alpha Go Zero: How and Why It Works

#106

Post by gwern »

cosminro wrote: post #104
gwern wrote: post #32

Earlier quoted context omitted.

> These are the two biggest distinguishing characteristics of AlphaGo Zero compared to previous AlphaGos, and the OP doesn't discuss either of them. David Silver disagrees. The most critical distinguishing characteristic is the expert/tree iteration which makes stable self-play possible at all.

A lot of people reading the paper miss this. I guess it's not emphasized enough. In the first paper, the selfplay trained policy is about 1500 in elo rating, while darkforest2 a supervised trained policy from Facebook is around the same, if not better. So selfplay wasn't of much use the first time around. While in the AlphaZero paper the selfplay trained policy has about 3000 elo rating.

> A lot of people reading the paper miss this. I guess it's not emphasized enough.

Yeah, it's hilariously underemphasized. 1 sentence, literally. Fortunately I was able to ask Silver directly and get confirmation that it's the tree iteration: https://www.reddit.com/r/MachineLearning/comments/76xjb5/ama...

Re: Alpha Go Zero: How and Why It Works

#107

Post by dvdkhlng »

gamegoblin wrote: post #97
nojvek wrote: post #91

Earlier quoted context omitted.

Or 100% rate of a draw. Like tic tac toe.

Due to komi, a go game cannot be drawn. That is, the 2nd player is awarded some number of points for the disadvantage of moving 2nd, and that number of points typically has a fractional 0.5 in it to break ties. The most common komi values these days are 6.5 or 7.5, depending on ruleset.

Depending on the rules, a game may be drawn due to an "eternal life" (an endless forced repetition).

https://senseis.xmp.net/?EternalLife

In Go this is very rare (as opposed to e.g. chess).

Re: Alpha Go Zero: How and Why It Works

#108

Post by javajosh »

dingo_bat wrote: post #59
javajosh wrote: post #3

"On a long enough timeline, everything is a discrete game." (With apologies to _Fight Club_) Personally, I look forward to the day when the software I own works for me to the extent of optimizing the decisions I make during the day, even many mundane ones. Properly executed, such a system could make a big difference in my quality of life. I believe that a big piece that is missing is a solid life-model, a life-repres…

I don't know why you're downvoted. It's an interesting thing to think about. Personally, I'd be eager to try it out for a year or so. I set the goal, my computer tells me what to do.

I seem to get a lot of upvotes, and then a lot of downvotes all in a flurry. I'd be curious to look at the data, to be honest, to see if it's a cohort. (No idea why they'd target me in particular)

Re: Alpha Go Zero: How and Why It Works

#109

Post by erikb »

gwern wrote: post #84
erikb wrote: post #83

I don't get what is new in the set of attributes that this article describes. Monte Carlo was already used in 2005 in AIs playing on KGS. Gradient Descent is a basic algorithm is a basic algorithm that I saw in an AI class in ~2008 as well. I bet both are even a lot older and well known by all experts. This is not what makes AlphaGo special or Zero successful. The curious thing about Zero is that usually with Gradien…

> The curious thing about Zero is that usually with Gradient Descent you run a huge risk of running into a local maximum and then stop evolving because every evolution makes you not better than the current step. No. The curious thing is that you can train a godawful huge NN with 40 layers via pure self-play with no checkpoints or baselines or pretraining or library of hard problems or any kind of stabilization mechan…

> divergence problems where the training explodes and engages in catastrophic forgetting

This is a new problem I haven't heard about. Thanks for adding it to the discussion.

> Er, this is exactly what OP is all about: the Monte Carlo tree search supervision.

And here, I don't know. I think I exclaimed confidently that this is a super old algorthim. What is new about it, that you need to mention it again? Really, may be that I'm misinterpreting something, or forgetting something, but the algorithm itself I'm quite sure I discussed with Go AI scientists in 2005 who had them in all their bots, afaik. Please correct if you believe my memory is cheating me here.

Re: Alpha Go Zero: How and Why It Works

#110

Post by gwern »

erikb wrote: post #109
gwern wrote: post #84

Earlier quoted context omitted.

> The curious thing about Zero is that usually with Gradient Descent you run a huge risk of running into a local maximum and then stop evolving because every evolution makes you not better than the current step. No. The curious thing is that you can train a godawful huge NN with 40 layers via pure self-play with no checkpoints or baselines or pretraining or library of hard problems or any kind of stabilization mechan…

> divergence problems where the training explodes and engages in catastrophic forgetting This is a new problem I haven't heard about. Thanks for adding it to the discussion. > Er, this is exactly what OP is all about: the Monte Carlo tree search supervision. And here, I don't know. I think I exclaimed confidently that this is a super old algorthim. What is new about it, that you need to mention it again? Really, may…

> but the algorithm itself I'm quite sure I discussed with Go AI scientists in 2005 who had them in all their bots, afaik

You did not because first, MCTS was only published by Coulom in 2006 (so they couldn't 'all' have been using it in 2005), and second because you are missing the crucial iteration between the heavy heuristic (NN) for playouts and the refined estimates from the playouts: https://news.ycombinator.com/item?id=15627834 Any MCTS in 2006 for Go would either be using random light playouts or simple unchanging hand-engineered heavy heuristics, hence, no possible recursive self-improvement in self-play.

Post reply on HN