DeepMind seems to be building the Wintermute to OpenAI’s Neuromancer. Where’s Turing?
MuZero: Mastering Go, chess, shogi and Atari without rules
41–50 of 79 posts
Re: MuZero: Mastering Go, chess, shogi and Atari without rules
#42Earlier quoted context omitted.
I've noticed all the top performing AI reinforcement algorithms i hear about know next to nothing about the initial rules. And not only do they perform as well as more supervised methods, but much better The one exception is self driving. I listened to the Lex Fridman interview with ceo of waymo recently and he made a case for the controlled environment (e.g. separate detection from decision making and planning) and…
A machine controlling a real car gathers feedback no faster than real time and cannot afford to learn the meanings of street signs from the consequences of ignoring them. A machine can learn the rules of Atari games from scratch by playing them orders of magnitude faster than real time and treating "death" as one signal among many. In order for a machine to learn driving the same way it learns Atari games, it seems t…
If you could make a virtual environment that was that good, you would have already solved the self driving problem.
I wonder if Google would be willing to pay people to add cameras to their cars to collect real world data in far larger scale.
Re: MuZero: Mastering Go, chess, shogi and Atari without rules
#43The full Atari game list (Appendix I) is interesting. It's not better at every game, and scores a zero on Pitfall and Montezuma's Revenge.
Re: MuZero: Mastering Go, chess, shogi and Atari without rules
#44Whoa, this is extremely impressive. Quotes from the BBC article: > "For the first time, we actually have a system which is able to build its own understanding of how the world works, and use that understanding to do this kind of sophisticated look-ahead planning that you've previously seen for games like chess. > "[It] can start from nothing, and just through trial and error both discover the rules of the world and u…
What does it mean to “not be given the rules”? If you set a child down in front of a chess board with the pieces nearby and they are not aware of the rules, I doubt they’d ever figure out how to play even a single correct game of chess. Heck, the child may decide to put the pieces in their mouth or dress them up as make belief characters. Without any concept of the rules you have no way of even knowing that you’ve se…
The point is this
> AlphaZero used the set of legal actions obtained from the simulator to mask the policy network at interior nodes. MuZero does not perform any masking within the search tree, but only masks legal actions at the root of the search tree where the set of available actions is directly observed. The policy network rapidly learns to exclude actions that are unavailable, simply because they are never selected.
MuZero still masks legal moves, but only at the root. All its parts are eventually trained on the output of its root, and so learn the legal moves.
The justify this root level masking by how the Atari will only allow you to perform legal moves, while a weak enough player may consider illegal moves while planning in your head.
The main thing that's slightly "hidden under the rug" is that for "masking" to make sense in the first place, MuZero needs to know a set of all moves that may be legal at some point in the games.
Re: MuZero: Mastering Go, chess, shogi and Atari without rules
#45Earlier quoted context omitted.
What does it mean to “not be given the rules”? If you set a child down in front of a chess board with the pieces nearby and they are not aware of the rules, I doubt they’d ever figure out how to play even a single correct game of chess. Heck, the child may decide to put the pieces in their mouth or dress them up as make belief characters. Without any concept of the rules you have no way of even knowing that you’ve se…
The point is this > AlphaZero used the set of legal actions obtained from the simulator to mask the policy network at interior nodes. MuZero does not perform any masking within the search tree, but only masks legal actions at the root of the search tree where the set of available actions is directly observed. The policy network rapidly learns to exclude actions that are unavailable, simply because they are never sele…
Oh okay. So it's a technique that essentially allows the tree search to be less "pedantic" about the rules in future states. Very interesting.
I would love to see how this might go for more complicated games such as NES adventure games and RPGs.
Re: MuZero: Mastering Go, chess, shogi and Atari without rules
#46Earlier quoted context omitted.
A machine controlling a real car gathers feedback no faster than real time and cannot afford to learn the meanings of street signs from the consequences of ignoring them. A machine can learn the rules of Atari games from scratch by playing them orders of magnitude faster than real time and treating "death" as one signal among many. In order for a machine to learn driving the same way it learns Atari games, it seems t…
If you could make a virtual environment that was that good, you would have already solved the self driving problem. I wonder if Google would be willing to pay people to add cameras to their cars to collect real world data in far larger scale.
That might be happening now already. I think Google sourced their dataset from dashcam videos uploaded to YouTubr?
Re: MuZero: Mastering Go, chess, shogi and Atari without rules
#47Earlier quoted context omitted.
What does it mean to “not be given the rules”? If you set a child down in front of a chess board with the pieces nearby and they are not aware of the rules, I doubt they’d ever figure out how to play even a single correct game of chess. Heck, the child may decide to put the pieces in their mouth or dress them up as make belief characters. Without any concept of the rules you have no way of even knowing that you’ve se…
The point is this > AlphaZero used the set of legal actions obtained from the simulator to mask the policy network at interior nodes. MuZero does not perform any masking within the search tree, but only masks legal actions at the root of the search tree where the set of available actions is directly observed. The policy network rapidly learns to exclude actions that are unavailable, simply because they are never sele…
> while a weak enough player may consider illegal moves while planning in your head
This isn't just weak players. E.g. strong chess players often consider moves as if blocking pawns weren't there. They might consider a bishop to be on a strong diagonal despite there being a blocking pawn because they can imagine moves that would happen if that pawn would disappear.
Re: MuZero: Mastering Go, chess, shogi and Atari without rules
#48As impractical as the idea is, reinforcement learning is so damn fun. I highly recommend others to play around with it. I originally was using the famous fork of OpenAi baseline, stable baseline but had issues with tuning with Optuna. I recently stumbled across Ray from Berkley [1] and it has a newer and fancier built-in hyper-parameter tuner. Even as a hardware engineer that's only a software hobbyist can make the c…
Eh, I'd say it's fun if you have a couple thousand TPUs lying around.
If you're just messing around with 1 GPU and a desktop PC you should be happy to get Atari breakout to work.
Re: MuZero: Mastering Go, chess, shogi and Atari without rules
#49I watched the Alpha Go vs. Lee Sedol games live. Big fan. That said, I think Deep Mind should go all in for solving practical real world problems.
I mean they pretty much solved protein folding this year...
Re: MuZero: Mastering Go, chess, shogi and Atari without rules
#50As impractical as the idea is, reinforcement learning is so damn fun. I highly recommend others to play around with it. I originally was using the famous fork of OpenAi baseline, stable baseline but had issues with tuning with Optuna. I recently stumbled across Ray from Berkley [1] and it has a newer and fancier built-in hyper-parameter tuner. Even as a hardware engineer that's only a software hobbyist can make the c…
Eh, I'd say it's fun if you have a couple thousand TPUs lying around. If you're just messing around with 1 GPU and a desktop PC you should be happy to get Atari breakout to work.
Definitely the worse part of RL is that it takes so long to train. But it works surprisingly well on Google Colab or equivalent.