Point this thing at the stock market and see how that game plays out.
MuZero: Mastering Go, chess, shogi and Atari without rules
11–20 of 79 posts
Re: MuZero: Mastering Go, chess, shogi and Atari without rules
#12Earlier quoted context omitted.
I think it would just smoke you from the outset. As far as I know it doesn't have a structured intelligence it can scale back - it would make the optimal move every time, destroying you like it destroyed top-tier players. I tried learning Go a little while back but hit a wall. Was thinking about trying this more gamified option: https://www.wolfeystudios.com/TheConquestOfGo.html
My rudimentary understanding of most reinforcement learning systems is that there is an "probability of optimality" associated with each action. Wouldn't there be a way to make the AI take the Nth optimal move or vary the degree of optimality with each move?
For these you are able to pick out worse moves than the optimal from them, but that's not actually the same thing as "play like a beginner, okay now play like an intermediate". These things are still openish questions and if nothing else there's a lot of room for improvement in tools to help you learn and review games.
Re: MuZero: Mastering Go, chess, shogi and Atari without rules
#13Amazing! Anyone has ideas on how to: 1. Bet on AGI, 2. Encourage AGI? I have a strong belief that it could grow and I’d like to contribute (and join the development)
It's not obvious this has much to do with AGI in the sense of human level sentience.
I think it definitely does. Is this AGI, or even close to it? No. But is it in the direction of AGI, or otherwise some small building block of it, that when worked on for decades, may contribute? Definitely
Re: MuZero: Mastering Go, chess, shogi and Atari without rules
#14Earlier quoted context omitted.
I think it would just smoke you from the outset. As far as I know it doesn't have a structured intelligence it can scale back - it would make the optimal move every time, destroying you like it destroyed top-tier players. I tried learning Go a little while back but hit a wall. Was thinking about trying this more gamified option: https://www.wolfeystudios.com/TheConquestOfGo.html
My rudimentary understanding of most reinforcement learning systems is that there is an "probability of optimality" associated with each action. Wouldn't there be a way to make the AI take the Nth optimal move or vary the degree of optimality with each move?
It turns out that taking a superhuman player and making it play like a weak human is surprisingly tricky. It's not so hard to make a weak player - you just take suboptimal moves instead of the best moves. But often these suboptimal moves are bizarre. A weak human chess player will lose their pieces as they fall prey to forks and skewers and so on - tricks that are hard to see coming for new player. But they will rarely actively throw a piece away by moving it into danger. Even a novice human is pretty decent at looking one move ahead. But to a chess engine, or an MuZero agent, a move that loses the queen immediately and a move that leads to a sequence that loses the queen in five turns are basically equal. And so an artificially-weak MuZero agent, or an artificially-weak Stockfish agent will tend to make 'mistakes' that not even a weak human would make. This makes them a little difficult to learn from.
There does exist research on how to make a human-like weak player: https://arxiv.org/abs/2006.01855
The basic idea is to look at weak human games and try to predict when a mistake will be made. But I don't know if there's any approach that can do that without access to a corpus of human errors.
Re: MuZero: Mastering Go, chess, shogi and Atari without rules
#15Is there a way to learn Go from scratch using these AIs? I wonder if it would pay off in the long run to be fully trained by one.
What you can do is checkout the algorithm at particular stages of development. AlphaZero&Friends start out not being very good at the game, then over time they learn and eventually become super human. You typically checkpoint the weights for the model at various stages. So early on, the algo would be like a 600 elo player for chess and then eventually get to superhuman elo levels. If you wanted to train using an AlphaX algo, you can gradually play against underdeveloped versions of the algo until you can beat them by loading up the weights at increasing stages of deveopment.
If you're curious how it would work, I implemented AlphaZero (but not Mu yet) using GBDTs instead of NNs here: https://github.com/cgreer/alpha-zero-boosted. Instead of saving the "weights" for a GBDT, you save the split points for the value/policy model trees, but the concept is the same.
Re: MuZero: Mastering Go, chess, shogi and Atari without rules
#16Whoa, this is extremely impressive. Quotes from the BBC article: > "For the first time, we actually have a system which is able to build its own understanding of how the world works, and use that understanding to do this kind of sophisticated look-ahead planning that you've previously seen for games like chess. > "[It] can start from nothing, and just through trial and error both discover the rules of the world and u…
I've noticed all the top performing AI reinforcement algorithms i hear about know next to nothing about the initial rules. And not only do they perform as well as more supervised methods, but much better The one exception is self driving. I listened to the Lex Fridman interview with ceo of waymo recently and he made a case for the controlled environment (e.g. separate detection from decision making and planning) and…
A machine controlling a real car gathers feedback no faster than real time and cannot afford to learn the meanings of street signs from the consequences of ignoring them. A machine can learn the rules of Atari games from scratch by playing them orders of magnitude faster than real time and treating "death" as one signal among many.
In order for a machine to learn driving the same way it learns Atari games, it seems that it would need an extremely high fidelity virtual environment to learn in. The high fidelity requirement would necessitate a lot of up-front investment in trying to get the simulations right. You might spend a whole career just trying to build a drivable Virtual Philadelphia as challenging as the real thing. The details would also make it much more expensive to run training sessions at high multiples of real time.
Given those factors, I'm not surprised that self-driving vehicle experiments just use real environments and don't try to learn the fundamental rules from scratch. But it's an interesting point that these choices may make it harder for agents to keep improving.
Re: MuZero: Mastering Go, chess, shogi and Atari without rules
#17Earlier quoted context omitted.
I've noticed all the top performing AI reinforcement algorithms i hear about know next to nothing about the initial rules. And not only do they perform as well as more supervised methods, but much better The one exception is self driving. I listened to the Lex Fridman interview with ceo of waymo recently and he made a case for the controlled environment (e.g. separate detection from decision making and planning) and…
A machine controlling a real car gathers feedback no faster than real time and cannot afford to learn the meanings of street signs from the consequences of ignoring them. A machine can learn the rules of Atari games from scratch by playing them orders of magnitude faster than real time and treating "death" as one signal among many. In order for a machine to learn driving the same way it learns Atari games, it seems t…
There's a middle ground too which alpha go leveraged which bootstrapped learning from actual human gameplay predicting user actions. That's what comma.ai does and AlphaGo still performed significantly better than explicit rules or higher level abstractions. That and a mix of simulations might yield better results
Re: MuZero: Mastering Go, chess, shogi and Atari without rules
#18Earlier quoted context omitted.
My rudimentary understanding of most reinforcement learning systems is that there is an "probability of optimality" associated with each action. Wouldn't there be a way to make the AI take the Nth optimal move or vary the degree of optimality with each move?
For these you are able to pick out worse moves than the optimal from them, but that's not actually the same thing as "play like a beginner, okay now play like an intermediate". These things are still openish questions and if nothing else there's a lot of room for improvement in tools to help you learn and review games.
You can make KataGo play moves that keep the score roughly even since it has a trained score head, e.g. kataJigo [1]. This will keep the game even to your level, a nice way to train.
Re: MuZero: Mastering Go, chess, shogi and Atari without rules
#19Same topic as a year ago, but deserves much more examination than it got then.
Yeah it's basically the exact same thing as in Nov 2019 right?
They're just hyping up their Nature publication. Or did I miss something?
Re: MuZero: Mastering Go, chess, shogi and Atari without rules
#20Amazing! Anyone has ideas on how to: 1. Bet on AGI, 2. Encourage AGI? I have a strong belief that it could grow and I’d like to contribute (and join the development)
It's not obvious this has much to do with AGI in the sense of human level sentience.
AGI is the main goal of DeepMind. They try to mimic human planning by finding similar strategies to what the human brain does, although of course there's never guarantee that their way is the right way.