Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

MuZero: Mastering Go, chess, shogi and Atari without rules

deepmind.com

Post reply on HN
Print view View original

61–70 of 79 posts

Re: MuZero: Mastering Go, chess, shogi and Atari without rules

#61

Post by martamorena284 »

philipkglass wrote: post #16
bko wrote: post #9

Earlier quoted context omitted.

I've noticed all the top performing AI reinforcement algorithms i hear about know next to nothing about the initial rules. And not only do they perform as well as more supervised methods, but much better The one exception is self driving. I listened to the Lex Fridman interview with ceo of waymo recently and he made a case for the controlled environment (e.g. separate detection from decision making and planning) and…

A machine controlling a real car gathers feedback no faster than real time and cannot afford to learn the meanings of street signs from the consequences of ignoring them. A machine can learn the rules of Atari games from scratch by playing them orders of magnitude faster than real time and treating "death" as one signal among many. In order for a machine to learn driving the same way it learns Atari games, it seems t…

Really not true. We have all the tools available... Just model a city in Unreal Engine 4 (GTA 6 anyone?). Your sensors, like LIDAR could do ray-queries, cameras could render views, etc. It's not 100% real, but probably real enough to learn the basics. Photorealistic graphics should be sufficient for AI to learn real-world interaction. In the end our whole world might be just that, a simulation. So I see zero reason why we couldn't train a self-driving car AI in one.

And the coolest thing is that this could also be used as a basis for a AAA video game, and these tend to make billions as well these days, so it's a win win for everyone. AI companies with funding should invest heavily into virtual reality and gaming, because they will need to perfect this to train their models.

Re: MuZero: Mastering Go, chess, shogi and Atari without rules

#62

Post by samatman »

ars wrote: post #42
philipkglass wrote: post #16

Earlier quoted context omitted.

A machine controlling a real car gathers feedback no faster than real time and cannot afford to learn the meanings of street signs from the consequences of ignoring them. A machine can learn the rules of Atari games from scratch by playing them orders of magnitude faster than real time and treating "death" as one signal among many. In order for a machine to learn driving the same way it learns Atari games, it seems t…

If you could make a virtual environment that was that good, you would have already solved the self driving problem. I wonder if Google would be willing to pay people to add cameras to their cars to collect real world data in far larger scale.

I do believe that's the basis of Tesla's play in the space: a bet that enough cameras collecting real-world data can beat out a dedicated from-scratch self-driving system in deployment.

Whether this will work remains very much an open question.

Re: MuZero: Mastering Go, chess, shogi and Atari without rules

#63

Post by klipt »

thomasahle wrote: post #44
chongli wrote: post #33

Earlier quoted context omitted.

What does it mean to “not be given the rules”? If you set a child down in front of a chess board with the pieces nearby and they are not aware of the rules, I doubt they’d ever figure out how to play even a single correct game of chess. Heck, the child may decide to put the pieces in their mouth or dress them up as make belief characters. Without any concept of the rules you have no way of even knowing that you’ve se…

The point is this > AlphaZero used the set of legal actions obtained from the simulator to mask the policy network at interior nodes. MuZero does not perform any masking within the search tree, but only masks legal actions at the root of the search tree where the set of available actions is directly observed. The policy network rapidly learns to exclude actions that are unavailable, simply because they are never sele…

Seems you could equivalently treat rule breaking as a loss, and any algorithm sophisticated enough to learn how to win will also learn to avoid breaking the rules.

Re: MuZero: Mastering Go, chess, shogi and Atari without rules

#64

Post by llamaz »

chongli wrote: post #33
ignoranceprior wrote: post #3

Whoa, this is extremely impressive. Quotes from the BBC article: > "For the first time, we actually have a system which is able to build its own understanding of how the world works, and use that understanding to do this kind of sophisticated look-ahead planning that you've previously seen for games like chess. > "[It] can start from nothing, and just through trial and error both discover the rules of the world and u…

What does it mean to “not be given the rules”? If you set a child down in front of a chess board with the pieces nearby and they are not aware of the rules, I doubt they’d ever figure out how to play even a single correct game of chess. Heck, the child may decide to put the pieces in their mouth or dress them up as make belief characters. Without any concept of the rules you have no way of even knowing that you’ve se…

Interestingly enough, this is exactly how one of the world chess champions, Jose Raul Capablanca, was said to have learned chess as a child.

It may be true, or perhaps it was a story concocted in order to emphasize his innate talent.

Re: MuZero: Mastering Go, chess, shogi and Atari without rules

#65

Post by kadoban »

tromp wrote: post #27
kadoban wrote: post #10

Earlier quoted context omitted.

You can play against open source reimplementations of some of the ideas behind AlphaGo family AIs. LeelaZero was one of the early ones, KataGo is probably your best bet right now. Sai is also in the mix. All are _very_ strong. KataGo is ungodly strong, it beats pros. Learning Go is about more than just playing against strong players, but it could help. The biggest difficulty is that the strong AIs aren't actually tha…

> strong AIs aren't actually that good at playing handicap games You wouldn't know that from watching JBXKataBot playing on KGS with a typical 5-7 stone handicap.

For the use being discussed (teaching beginners), you really need 9 stones, and it needs to work well. 7 stones is getting close though. I'll take a look at those games.

Last I saw I remember katago playing up to maybe 4 stones pretty well but the games being poorer quality over that.

Re: MuZero: Mastering Go, chess, shogi and Atari without rules

#66

Post by kadoban »

_hark wrote: post #18
kadoban wrote: post #12

Earlier quoted context omitted.

For these you are able to pick out worse moves than the optimal from them, but that's not actually the same thing as "play like a beginner, okay now play like an intermediate". These things are still openish questions and if nothing else there's a lot of room for improvement in tools to help you learn and review games.

You can make KataGo play moves that keep the score roughly even since it has a trained score head, e.g. kataJigo [1]. This will keep the game even to your level, a nice way to train. [1] https://github.com/sanderland/katrain#ais

I'm not sure that's a good idea at all for training, though it is a really neat trick.

For training you really want your good moves to be rewarded and your bad moves pointed out, but if the AI just plays up or down to match what you do instead, there's no signal getting back to you on how you're doing.

Re: MuZero: Mastering Go, chess, shogi and Atari without rules

#67

Post by dwohnitmok »

Buttons840 wrote: post #60

I'd still be most impressed to see an AI beat the top Civilization players. No mechanical advantage since it's turn based, but there are several different types of decisions to make beyond just "move a piece". AIs haven't yet conquered such environments. It would also give the gaming industry a kick in the pants to start making better AIs.

I am fairly confident a team of DeepMind's calibre could put together an AI in fairly short order that would demolish top-level Civilization players. Despite my confidence, I still would love to see such a thing made.

DeepMind made a good effort with AlphaStar at building an AI that could compete with top-level humans in Starcraft. It wasn't superhuman; it could still be consistently beaten by the absolute best Starcraft players, especially as Zerg or Terran. However, as Protoss, AlphaStar was truly a pro-level player. I'm somewhat surprised DeepMind didn't go further and try to optimize AlphaStar to truly be superhuman. I'm not sure if that indicates a fundamental limitation of their approach or whether it was a shift in approach. This was with successively refined limitations on AI action speeds that caused AlphaStar to really rely on strategy and tactics rather than brute force speed.

Regardless, real-time strategy games feel much more difficult than turn-based strategy games to develop a good AI for. Just being able to split things into discrete turns seems like a massive simplification.

Re: MuZero: Mastering Go, chess, shogi and Atari without rules

#69

Post by ngcc_hk »

We live only once. Could this uniqueness meant some of life decision must be done without repeating the case billion of time, which obviously s impossible.

I read qm. But does this actually useful for partial info. This is also another life situation where you never have full information.

I still wonder about the intelligence.

Re: MuZero: Mastering Go, chess, shogi and Atari without rules

#70

Post by thomasahle »

nightcracker wrote: post #47
thomasahle wrote: post #44

Earlier quoted context omitted.

The point is this > AlphaZero used the set of legal actions obtained from the simulator to mask the policy network at interior nodes. MuZero does not perform any masking within the search tree, but only masks legal actions at the root of the search tree where the set of available actions is directly observed. The policy network rapidly learns to exclude actions that are unavailable, simply because they are never sele…

> while a weak enough player may consider illegal moves while planning in your head This isn't just weak players. E.g. strong chess players often consider moves as if blocking pawns weren't there. They might consider a bishop to be on a strong diagonal despite there being a blocking pawn because they can imagine moves that would happen if that pawn would disappear.

I suppose you are right. But MuZero won't be able to do this, since it's training forces it to consider legal moves in its planning.
Post reply on HN