AlphaGo Zero: Learning from scratch • Hacker News Board

Re: AlphaGo Zero: Learning from scratch

#121

Post by gizmo686 » Wed, Oct 18, 2017, 8:57 PM UTC

alex_stoddard wrote: post #114

kndyry wrote: post #104

Earlier quoted context omitted.

I don't think it's an overstatement to say that, since playing Lee Sedol in 2016, AlphaGo has completely revolutionized professional and amateur go. It's certainly not unprecedented — the last major revolution happened in the early 20th century (often called the 'Shin Fuseki' era [0]) — but AlphaGo has demonstrably surpassed any previous high-water mark. > I wonder if this system produced more new styles of play. Abs…

I have only skimmed the paper but one thing I don't see any discussion of is whether komi (the handicap given to white for going second) is correct. They do say the rules used for all games, including self-play, set komi consistently to 7.5 . If the strongest AI was consistently winning predominantly with one color it would be an indication that komi isn't fair for the best play. Of the 20 games released for the stro…

You can only change komi by full point increments. There is a .5 to break ties, but a komi of 7.5 is identical to one of 7.4.

From a theoretical standpoint, any non-integer komi should lead to one player winning 100% of the time. So even if the actual win ratio is 14:6 at komi=7.5 that might still be the best value.

Re: AlphaGo Zero: Learning from scratch

#122

Post by psb217 » Wed, Oct 18, 2017, 8:58 PM UTC

shmageggy wrote: post #74

greysphere wrote: post #45

Earlier quoted context omitted.

TD-gammon is a well known version of this technique (with 2 ply lookahead, vs a 1600 deep mcts) https://en.m.wikipedia.org/wiki/TD-Gammon Temporal difference learning was previously consider weak at 'tactical' games, ie ones with gamestates that require long chains of precise moves to improve position (like many checkmate scenarios in chess) . For anyone more familiar with this technique, is it clear how the mcts/che…

Yeah it's not clear to me why temporal difference learning all of a sudden works so well here? Is it the case that nobody had really tried it for learning a policy for Go with a strong NN architecture? In the Methods they mention TD learning for value functions but I don't see anything about policies. edit: OK, they're calling it policy iteration as opposed to TD learning. I guess I don't get the difference.

TD learning is, in some sense, a component of policy iteration. TD learning is about learning the value function for a given policy. In policy iteration you use a value function to decide how to update the policy for which the value function was estimated, and you iterate between the "learn value" and "update policy" steps.

https://www.cs.cmu.edu/afs/cs/project/jair/pub/volume4/kaelb...

Re: AlphaGo Zero: Learning from scratch

#123

Post by aeleos » Wed, Oct 18, 2017, 8:59 PM UTC

ionforce wrote: post #108

etiam wrote: post #102

Earlier quoted context omitted.

Please don't refer to them as black boxes. The internals are fully observable.

Is there meaningful information in what one observes?

Yes, it turns out you can find meaningful information. etiam provided this https://arxiv.org/pdf/1312.6034.pdf The main issue is making sure what you are looking for is actually what the network is doing. You have to correctly interpret and visualize a jumble of numbers, which usually requires a hypothesis about how it worked in the first place. But assuming both go well you can gain meaningful information.

Re: AlphaGo Zero: Learning from scratch

#124

Post by grondilu » Wed, Oct 18, 2017, 9:02 PM UTC

> If similar techniques can be applied to other structured problems, such as protein folding, reducing energy consumption or searching for revolutionary new materials,

Protein folding sounds like a nice idea for their next challenge.

Re: AlphaGo Zero: Learning from scratch

#125

Post by Symmetry » Wed, Oct 18, 2017, 9:03 PM UTC

titzer wrote: post #2

This is pretty incredible, especially the power dissipation results. Only 4 TPUs? Humans are toast.

That's still 10 times as much energy as a human body or 100 times as much as a human brain. But yeah, it's not like they're throwing a datacenter at this.

Re: AlphaGo Zero: Learning from scratch

#126

Post by jononor » Wed, Oct 18, 2017, 9:04 PM UTC

Florin_Andrei wrote: post #25

galkk wrote: post #21

Why don't use the same approach for chess? It's very interesting to see if it is able to handle much more advanced and tuned engines that exist for chess, game with considerable much more complicated rules?

I think chess is less compelling because, in a sense, it is a "solved problem" - superhuman AI chess players already exist. And chess, while it does have more complex base rules, has a much lower combinatorial complexity than Go.

I think that the existence of highly optimized chess AI makes it interesting from two angles: 1) Generalization: Can one make AI using same approach that can play both chess and Go at superhuman levels 2) Efficiency: Can these newer methods match or outperform also in terms of compute/energy costs

But maybe not sexy enough, or we just don't hear about it as much.

Re: AlphaGo Zero: Learning from scratch

#127

Post by Jach » Wed, Oct 18, 2017, 9:04 PM UTC

alex_stoddard wrote: post #114

kndyry wrote: post #104

Earlier quoted context omitted.

I don't think it's an overstatement to say that, since playing Lee Sedol in 2016, AlphaGo has completely revolutionized professional and amateur go. It's certainly not unprecedented — the last major revolution happened in the early 20th century (often called the 'Shin Fuseki' era [0]) — but AlphaGo has demonstrably surpassed any previous high-water mark. > I wonder if this system produced more new styles of play. Abs…

I have only skimmed the paper but one thing I don't see any discussion of is whether komi (the handicap given to white for going second) is correct. They do say the rules used for all games, including self-play, set komi consistently to 7.5 . If the strongest AI was consistently winning predominantly with one color it would be an indication that komi isn't fair for the best play. Of the 20 games released for the stro…

I don't remember where I read it but in some earlier versions of AlphaGo they tried a komi of 6.5 and black ended up winning more often. That indicates the correct komi value is 7, but since Go doesn't have ties, you have to pick which side you want to favor to break the tie. (White seems reasonable.)

Re: AlphaGo Zero: Learning from scratch

#128

Post by thefalcon » Wed, Oct 18, 2017, 9:04 PM UTC

grondilu wrote: post #124

> If similar techniques can be applied to other structured problems, such as protein folding, reducing energy consumption or searching for revolutionary new materials, Protein folding sounds like a nice idea for their next challenge.

https://www.bloomberg.com/news/articles/2017-10-18/deepmind-...

Indeed.

Re: AlphaGo Zero: Learning from scratch

#129

Post by mtremsal » Wed, Oct 18, 2017, 9:05 PM UTC

shmageggy wrote: post #71

Looks like the performance improvement comes from two key ingredients: 1) Using Residual networks instead of normal convolutional layers 2) Using a smarter policy training loss that uses the full information from a MCTS at each move. In the previous version, I believe they just ran the policy network to the end of the game and used a very weak {0, 1} reinforcement signal over all of the moves played. Here, it looks l…

How is it different to apply the loss on each actual move at the end of the game VS on each rollout (which is itself a tiny game)? Does it help reinforce learning towards the end game as shorter rollouts are needed? Is the more accurate information then propagated to earlier moves as well?

Re: AlphaGo Zero: Learning from scratch

#130

Post by aeleos » Wed, Oct 18, 2017, 9:11 PM UTC

andbberger wrote: post #98

aeleos wrote: post #88

Earlier quoted context omitted.

That is really interesting. Given a neural network that solely exist to play Go, one that is influenced by the human mind is limited compared to the exact same set of neurons that doesn't have that influence. EDIT: changed a set of neurons to neural network per andbbergers comments

Please don't refer to it as 'a set of neurons' - it only serves to fuel the (IMO) absolutely ridiculous AI winter fearmongering, and is also just a bad description. Neural nets are linear algebra blackboxes, the connections to biology are tenuous at best. Sorry to be that guy, but the AI hype is getting out hand. COSYNE this year was packed with papers comparing deep learning to the brain... it drives me nutty. Convn…

I changed my comment to neural network since a set of neurons is somewhat wrong, but I don't really agree that there isn't much of a connection between this and biology. While there might not be much of a connection between how they currently work and how our brains work, the whole point of machine learning and neural networks is to improve computers performance on the things we are good at. And while originally it was loosely modeled on it, and might be different know, it doesn't make it so people can't compare it to the brain. It would be wrong to say it is exactly like the brain, but I don't think there is anything wrong with comparing and contrasting the two. If our goal is to improve performance and we are the benchmark, then why shouldn't we compare them.

What I found interesting was mainly that it was us who nudged the parameter space you talked about into the "wrong" one manifold, especially given how old and complicated Go is. The sheer amount of human brain power that has been put into getting good at a game wasn't able to find certain aspects of it, and in 60 hours of training a neural network was able to.