Live data from Hacker News

Alpha Go Zero: How and Why It Works

tim.hibal.org

61–70 of 116 posts

Re: Alpha Go Zero: How and Why It Works

#61
post #3

"On a long enough timeline, everything is a discrete game." (With apologies to _Fight Club_) Personally, I look forward to the day when the software I own works for me to the extent of optimizing the decisions I make during the day, even many mundane ones. Properly executed, such a system could make a big difference in my quality of life. I believe that a big piece that is missing is a solid life-model, a life-repres…

This may interest you, AI as an artificial belief system. http://us1.campaign-archive.com/?u=78cbbb7f2882629a5157fa593...

My interpretation is that AI can allow you to rapidly load and unload mental models so that you're free to devote 100% of your mind to what only you (a human) can do.

Re: Alpha Go Zero: How and Why It Works

#62
They are two things a human brain does when playing chess or go: evaluating a position and mentally playing some positions (by doing a search tree).

The AlphaGo neural network is able to do the first part (evaluating positions) but the search tree is still a hand crafted algorithm. Do they have plans to work on a version with a pure neural network? (i.e. a version which would be able to learn how to do a search tree.)

Re: Alpha Go Zero: How and Why It Works

#63
post #26

The main reason AlphaGo Zero learns so much faster than its predecessors is because it uses temporal-difference learning.[1] This effectively removes a huge amount of the value network's state space for the learning algorithm to search through, since it bakes in the assumption that a move's value ought to equal that of the best available move in the following board position, which is exactly what you'd expect for a g…

How would OpenAI's self-play be different?

https://blog.openai.com/competitive-self-play/

Re: Alpha Go Zero: How and Why It Works

#64
post #26

The main reason AlphaGo Zero learns so much faster than its predecessors is because it uses temporal-difference learning.[1] This effectively removes a huge amount of the value network's state space for the learning algorithm to search through, since it bakes in the assumption that a move's value ought to equal that of the best available move in the following board position, which is exactly what you'd expect for a g…

For anyone interested: Learn more on TD and RL in general from Sutton (inventor of TD-lambda) and Barto's book: http://www.incompleteideas.net/sutton/book/the-book.html

Sidenote: It used to be that simply googling "sutton barto book" would bring you to the right place with the first suggested link. Now this stuff is so popular all of a sudden, I needed to consult the link I had set on my own page in order to find it. It's curious how the growth of popularity of an idea will with time obscure it's own roots and primary sources. On the plus side, TIL that Sutton's working on a 2nd edition! =)

Re: Alpha Go Zero: How and Why It Works

#65

Would be really cool to see a generic framework for this, where you can plug in the rules of your discrete-deterministic-game-with-perfect-information and get a superhuman bot. Does something like this already exist?

Check out the field of general game playing:

https://en.wikipedia.org/wiki/General_game_playing

http://www.ggp.org

http://logic.stanford.edu/ggp/readings/retrospective.html

Re: Alpha Go Zero: How and Why It Works

#66
post #47
post #46

Saw the AlphaGo movie at a festival recently. Been following the AlphaGo Zero developments, which leap-frog what was going on in the movie (although still very much worth seeing). One thing I was curious about is if Go would be considered solved, either hard or weakly solved, since AlphaGo Zero at this point doesn't seem to be able to be beat by any living human. Wikipedia does not list it as solved in either sense,…

if alpha go is just an adversarial network to brute force states, then it is not solved (note I don't research alphaGo, and most of what I know about it is from HN comments)

Oh so you’re an internet idiot, who shits on other people’s hard work even if completely clueless about it —- what’s it like over there?

Re: Alpha Go Zero: How and Why It Works

#67
post #26

The main reason AlphaGo Zero learns so much faster than its predecessors is because it uses temporal-difference learning.[1] This effectively removes a huge amount of the value network's state space for the learning algorithm to search through, since it bakes in the assumption that a move's value ought to equal that of the best available move in the following board position, which is exactly what you'd expect for a g…

Isn’t the assumption you mention simply the assumption made in the Bellmann equation? In that case the assumption would have been made in previous versions of AlohaGo too.

Re: Alpha Go Zero: How and Why It Works

#68
post #26

The main reason AlphaGo Zero learns so much faster than its predecessors is because it uses temporal-difference learning.[1] This effectively removes a huge amount of the value network's state space for the learning algorithm to search through, since it bakes in the assumption that a move's value ought to equal that of the best available move in the following board position, which is exactly what you'd expect for a g…

OP explicitly discusses the second big distinguishing in the opening paragraph of the section titled, 'The Alpha Zero Neural Ne':

“The Alpha Zero algorithm produces better and better expert policies and value functions over time by playing games against itself with accelerated Monte Carlo tree search. The expert policy π and the approximate value function Ŵ are both represented by deep neural networks. In fact, to increase efficiency, Alpha Zero uses one neural network f that takes in the game state and produces both the probabilities over the next move and the approximate state value. (Technically, it takes in the previous eight game states and an indicator telling it whose turn it is.)”

Regarding whether OP touches on temporal-difference learning I am unqualified to say but they do not explicitly mention it. Furthermore I am unqualified to judge how central this technique is to the level of play achieved. However in the DeepMind paper (pg. 20)† that start talking about temporal-difference learning thus:

“Self-play reinforcement learning has previously been applied to the game of Go. NeuroGo[40, 41] used a neural network to represent a value function, using a sophisticated architecture based on Go knowledge regarding connectivity, territory and eyes. This neural network was trained by temporal-difference learning[42] to predict territory in games of self-play, building on prior work[43]. A related approach, RLGO[44], represented the value function instead by a linear combination of features, exhaustively enumerating all 3 × 3 patterns of stones; it was trained by temporal-difference learning to predict the winner in games of self-play. Both NeuroGo and RLGO achieved a weak amateur level of play.”

I'm no expert but this implies to me that it was probably the sum total of all the subtle architectural decisions made by the DeepMind team plus their AI hardware and software platform that made AlphaGo Zero excel.

https://deepmind.com/documents/119/agz_unformatted_nature.pd...

Re: Alpha Go Zero: How and Why It Works

#69
post #3

"On a long enough timeline, everything is a discrete game." (With apologies to _Fight Club_) Personally, I look forward to the day when the software I own works for me to the extent of optimizing the decisions I make during the day, even many mundane ones. Properly executed, such a system could make a big difference in my quality of life. I believe that a big piece that is missing is a solid life-model, a life-repres…

What if everybody uses the same software by a handful of companies, maybe only one? It will be interesting to see how it resolves conflicts between goals of different customers. More interestingly, how does it regards the goals of its company compared to the ones of its customers?

Based on human nature and history I don't expect anything good coming from that.

Re: Alpha Go Zero: How and Why It Works

#70
Are there any plans to do this for Chess?

I imagine that this is an iteration of the Alpha Go engine, people working on this are very current with Alpha Go.

If Chess is similar, then wouldn't DeepMind be able to bootstrap game knowledge. Perhaps this isn't a big goal, but Chess is Chess after all.

Post reply on HN