51–60 of 84 posts
Given their track history, I don't think it's likely the DeepMind team are trying to be sneaky here. They'd be found out eventually given how big their claim is. When working in academia, I found it very common for research papers to not come with source code or enough information to allow you to replicate experiments yourself. You usually have to pester the author. I don't find the (valid) criticisms here that unusu…
To me, what follows does not seem to justify this claim, but it is not my field. In addition, some of his arguments seem to be beside the point - for example, he asks "Does AlphaZero completely learn from self-play?", and while saying generally, yes, he objects that encoding the rules was a non-trivial matter. While that certainly seems to be true, it does not seem to have much bearing on the claim that AlphaZero apparently learned to win through self-play. That, to me, seems to be its singular achievement: the absence of human-written tactics and strategy (unless the encoding of the rules somehow prefigured them, which is not being claimed here, and which seems highly unlikely.)
Given their track history, I don't think it's likely the DeepMind team are trying to be sneaky here. They'd be found out eventually given how big their claim is. When working in academia, I found it very common for research papers to not come with source code or enough information to allow you to replicate experiments yourself. You usually have to pester the author. I don't find the (valid) criticisms here that unusu…
It’s always been a trade off between using more expensive heuristics on fewer moves or cheaper heuristics on more moves. The “number of moves” thing only says that they went all-in on the better heuristics angle. In fact, there was something posted recently about test games where they played go by just using the first move suggested by the search heuristic and that was pretty strong.
Not sure how big breakthrough in Ai domain it is, but after watching some of the AplhaZero games against Stockfish (available at youtube) I'm convinced it's a true revolution in how computers play chess. It plays so much more like humans do, without the usual crazy micro management that other chess engines love to do. It's not boring to watch, makes very little weird moves that only a computer would ever make. If I d…
Can anyone summarise how self play works here if AlphaZero only starts out being told the rules of the game? Does it initially plays games using completely random moves as both players? Is it only told who the winner is with no other feedback? How is it able to learn e.g. that certain moves at the start eventually lead to a win?
1/ Initialise a random neural network.
2/ For any given position, use the neural network to estimate the probability of win (or draw).
3/ Play multiple branches out to the end, to get an estimate of the 'real' probability of win (for the current skill level)
4/ Modify the neural network so its estimate of the probability of win more closely matches the actual probability of win
5/ Goto 2 :)
Can anyone summarise how self play works here if AlphaZero only starts out being told the rules of the game? Does it initially plays games using completely random moves as both players? Is it only told who the winner is with no other feedback? How is it able to learn e.g. that certain moves at the start eventually lead to a win?
The value network is simply updated to match the real outcomes of games of self-play.
The policy network is updated to match the results of a tree search; for each board position many thousands of lines are explored using the value and policy networks, and then the policy is updated to match (a somewhat 'sharpened' version of) the number of lines in which each move was chosen.
When exploring each lines, at each step the move 'a' is chosen from the current board state 's' which maximizes Q(s, a) + P(s, a) / (1 + N(s, a)), where R is the policy network, Q is the average value network evaluation for lines where 'a' was picked from 's' (with the appropriate signs to match the current player at 's'), and N is the number of simulations in which 'a' was picked. When we reach an unseen board state, it is evaluated with the policy network and a new line is explored from the root.
This is less circular than it may seem because:
a) The value network is trained using real outcomes. b) Towards the end of the game, the tree search sees real outcomes.
This training procedure allows the network to 'bootstrap', learning progressively more complex knowledge about how to play effectively.
Can anyone summarise how self play works here if AlphaZero only starts out being told the rules of the game? Does it initially plays games using completely random moves as both players? Is it only told who the winner is with no other feedback? How is it able to learn e.g. that certain moves at the start eventually lead to a win?
It is my understanding that the system uses wins to learn to estimate the value of board positions: the position just before the win should be highly valued for the winner and inversely so for the loser. Going back further in time, there will be less confidence, but still a higher value for the winner.
* evaluate a position by playing lots of random games until the end using "likely" moves (in their order of likelihood). this gives you a "win rate" (i.e. empirical probability of winning) that is used as a position's score.
* train a neural net to predict that win rate based score (without having to play random games).
* use that neural net to predict which moves are "likely", thereby making the playing of random games more efficient, yielding more expressive scores, which in turn again improve the accuracy of the network.
* use the above described algorithm to self-play games, where every move is determined by looking at the win-rates given by the neural-net-driven quasi-random games, continuously training the score-estimating neural net.
This is just my high-level understanding without knowing too much about the topic. Monte Carlo Tree search is a little more complex than what I try to explain here.
Can anyone summarise how self play works here if AlphaZero only starts out being told the rules of the game? Does it initially plays games using completely random moves as both players? Is it only told who the winner is with no other feedback? How is it able to learn e.g. that certain moves at the start eventually lead to a win?
There is a value network which estimates the win rate for the current player from a given board state, and a policy network which estimates the probability that each move should be played. As of the more recent iterations, these networks share their bottom layers for greater computational and training efficiency. The value network is simply updated to match the real outcomes of games of self-play. The policy network…
Can anyone summarise how self play works here if AlphaZero only starts out being told the rules of the game? Does it initially plays games using completely random moves as both players? Is it only told who the winner is with no other feedback? How is it able to learn e.g. that certain moves at the start eventually lead to a win?
There is a value network which estimates the win rate for the current player from a given board state, and a policy network which estimates the probability that each move should be played. As of the more recent iterations, these networks share their bottom layers for greater computational and training efficiency. The value network is simply updated to match the real outcomes of games of self-play. The policy network…
> and a policy network which estimates the probability that each move should be played.
So for this network, the input is the before and after board state and the output is the probability that this move should be played?
It's like they don't understand the exponential nature of depth search in chess…