Mastering Chess and Shogi by Self-Play with General Reinforcement Learning

281–282 of 282 posts

Re: Mastering Chess and Shogi by Self-Play with General Reinforcement Learning

#281

Post by gwern » Wed, Dec 13, 2017, 1:40 AM UTC

robrenaud wrote: post #184

thom wrote: post #171

Earlier quoted context omitted.

I dunno, seems like Google would just do this instead of keep around the pesky neural net at runtime. There's an _awful_ lot of computation going on inside, and it's necessarily hugely interconnected. I'd be impressed if someone had already done it, but it seems a great avenue of research if not. I suppose it goes hand in hand with models for which you can actually _explain_ their results, which certainly is an activ…

There are well-known techniques that work pretty well to shrink neural nets a lot while keeping almost all of their performance. See Geoffrey Hinton's model distillation papers. The first AlphaGo paper had a system that used tons of computation, and was followed up by one that used much less and worked even better. Not speaking for Google, but I think it's a bit of a race to publish great results first. I wouldn't be…

A really good example of model distillation also comes from DM: their new realtime WaveNet used in Google Assistant. The first WaveNet was ungodly slow due to redundant computation; but even after that, it still was not realtime simply because the CNN is too deep and slow. But you need the CNN to be deep & big in order to train good audio generation. Model distillation to the rescue: take a wide fast small CNN and train it to imitate the slow deep WaveNet. Result: WaveNet quality realtime voice generation which can be deployed to the masses.

Re: Mastering Chess and Shogi by Self-Play with General Reinforcement Learning

#282

Post by gwern » Wed, Dec 13, 2017, 1:47 AM UTC

eref wrote: post #127

narrator wrote: post #113

So when are they going to apply this to Atari Games or well anything? The next step is they have one AI figure out the rules by making a GAN that imitates player behavior and the other AI be Alpha Go which tweaks the GAN inputs to generate different moves to win. Voila...Almost General Purpose AI that can learn to play any game.

The main problem is that we still lack good generative models and good ways of interrogating them. GANs are unstable and difficult to apply to time series, VAEs suffer from posterior collapse, WaveNet/PixelRNN grow with the input size and overemphasize the details, RNNs are hard to train because we lack good training algorithms. Generally, small errors tend to compound in step-wise predictions because NNs do not gene…

To expand on eref's comment a little: you absolutely could apply this or MCTS to ALE (and Guo et al 2014 did it very nicely). After all, the ALE is deterministic and simulatable by definition, so of course you can explore the game tree and reset the simulation as necessary. But people aren't much interested in this approach because using the ALE as a 'simulator' is cheating as far as testing full-strength AI techniques (we don't have simulators of the real world, after all), and the ALE games themselves (unlike Go) are of little intrinsic interest so there's no real benefit to engaging in cheating.

Post reply on HN