Live data from Hacker News

MuZero: Mastering Go, chess, shogi and Atari without rules

deepmind.com

51–60 of 79 posts

Re: MuZero: Mastering Go, chess, shogi and Atari without rules

#51
post #11

Point this thing at the stock market and see how that game plays out.

It's not so clear to me how you would train a reinforcement learning agent for the stock market. You have historic data for prices etc. But that's more of a supervised learning thing. You could set it loose on one of those realtime market simulators, but the agents actions wouldn't have any impact on the simulation right?

Re: MuZero: Mastering Go, chess, shogi and Atari without rules

#52
post #5

Is there a way to learn Go from scratch using these AIs? I wonder if it would pay off in the long run to be fully trained by one.

Yes, you can just `pip install katrain` followed by `python -m katrain` to get started. Personally I would recommend at least reading abut the rules first (unlike MuZero).

I think the strength or lack thereof of your opponent is actually much less important than the strength of the AI you use to review your games. After each game you should study the AI's advice and learn the moves it recommends.

Re: MuZero: Mastering Go, chess, shogi and Atari without rules

#55

The full Atari game list (Appendix I) is interesting. It's not better at every game, and scores a zero on Pitfall and Montezuma's Revenge.

You might be interested to read https://deepmind.com/blog/article/Agent57-Outperforming-the-... which has an alternative approach that does better on those games. Basically those games involve lots of “exploration” to find the winning states, so you need some algorithm that is incentivized to explore through a large state space even when it hasn’t seen any reward there.

Re: MuZero: Mastering Go, chess, shogi and Atari without rules

#56

As impractical as the idea is, reinforcement learning is so damn fun. I highly recommend others to play around with it. I originally was using the famous fork of OpenAi baseline, stable baseline but had issues with tuning with Optuna. I recently stumbled across Ray from Berkley [1] and it has a newer and fancier built-in hyper-parameter tuner. Even as a hardware engineer that's only a software hobbyist can make the c…

Eh, I'd say it's fun if you have a couple thousand TPUs lying around. If you're just messing around with 1 GPU and a desktop PC you should be happy to get Atari breakout to work.

The published hyperparameters are usually ridiculously conservative, for the simple games like breakout and pong you can usually converge in far fewer frames than in the papers.

Re: MuZero: Mastering Go, chess, shogi and Atari without rules

#57
post #11

Point this thing at the stock market and see how that game plays out.

Many very smart people have tried and failed. State of the art remains very basic supervised models with hand engineered features. In the markets, data is permanently scarce, so these methods don't work well. In the RL problems that DeepMind is solving, data is literally unlimited, and that's the problem space that these methods have been designed for.

Re: MuZero: Mastering Go, chess, shogi and Atari without rules

#58
post #11

Point this thing at the stock market and see how that game plays out.

It's not so clear to me how you would train a reinforcement learning agent for the stock market. You have historic data for prices etc. But that's more of a supervised learning thing. You could set it loose on one of those realtime market simulators, but the agents actions wouldn't have any impact on the simulation right?

There's two problems in markets, price prediction and execution (ie what to do with your prediction). The former is a supervised learning problem but the latter is an action space problem ie an RL problem. Although nobody in industry has gotten any RL methods too work, they overfit to the incredibly small data sets.

Re: MuZero: Mastering Go, chess, shogi and Atari without rules

#59
post #9

Whoa, this is extremely impressive. Quotes from the BBC article: > "For the first time, we actually have a system which is able to build its own understanding of how the world works, and use that understanding to do this kind of sophisticated look-ahead planning that you've previously seen for games like chess. > "[It] can start from nothing, and just through trial and error both discover the rules of the world and u…

I've noticed all the top performing AI reinforcement algorithms i hear about know next to nothing about the initial rules. And not only do they perform as well as more supervised methods, but much better The one exception is self driving. I listened to the Lex Fridman interview with ceo of waymo recently and he made a case for the controlled environment (e.g. separate detection from decision making and planning) and…

Minor tangent, but I think you’re referring to this video[1] with Dmitri Dolgov. He’s the CTO of Waymo, not CEO.

[1] https://youtu.be/P6prRXkI5HM

Re: MuZero: Mastering Go, chess, shogi and Atari without rules

#60
I'd still be most impressed to see an AI beat the top Civilization players. No mechanical advantage since it's turn based, but there are several different types of decisions to make beyond just "move a piece". AIs haven't yet conquered such environments.

It would also give the gaming industry a kick in the pants to start making better AIs.

Post reply on HN