DeepMind achieves SOTA image recognition with 8.7x faster training
1–10 of 85 posts
Re: DeepMind achieves SOTA image recognition with 8.7x faster training
#2> 8.7x faster to train
This is an achievement but it would be helpful to have put "to train" in the title as this is quite different from efficiency at inference time, which is what often actually matters in deployed applications.
From Table 3 on Page 7 it appears to me that NFNet is significantly heavier in the number of parameters than EfficientNet for similar accuracies. For example EffNet-B5 achieves 83.7% with 30M params and 9.9B FLOPs on Top-1 while NFNet-F0 achieves 83.6% with 71.5M params and 12.38B FLOPs on Top-1.
It appears to me at first glance that NFNet has not achieved SOTA at inference.
Re: DeepMind achieves SOTA image recognition with 8.7x faster training
#3I was skeptical of the 83%-whatever top-1 accuracy on Imagenet. But someone pointed out that the accuracy increases when the model is pretrained on JFT, google's proprietary 100-million image dataset, the model's accuracy increases to 87%-whatever.
That's pretty interesting. It implies that the original accuracy rating might be legit. The concern is that we're chasing the imagenet benchmark as if it's the holy grail, when in fact it's a very narrow slice of what we normally care about as ML researchers. However, the fact that pretraining on JFT increases the accuracy means that the model is generalizing, which is very interesting; it implies that models might be "just that good now."
Or more succinctly, if the result was bogus, you'd expect JFT pretraining to have no effect whatsoever (or a negative effect). But it has a positive result.
The other thing worth mentioning is that AJMooch seems to have killed batch normalization dead, which is very strange to think about. BN has had a long reign of some ~4 years, but the drawbacks are significant: you have to maintain counters yourself, for example, which was quite annoying.
It always seemed like a neural net ought to be able to learn what BN forces you to keep track of. And AJMooch et al seem to prove this is true. I recommend giving evonorm-s a try; it worked perfectly for us the first time, with no loss in generality, and it's basically a copy-paste replacement.
(Our BigGAN-Deep model is so good that I doubt you can tell the difference vs the official model. It uses AJMooch's evonorm-s rather than batchnorm: [1] https://i.imgur.com/sfGVbuq.png [2] https://i.imgur.com/JMJ1Ll0.png and lol at the fake speedometer.)
Re: DeepMind achieves SOTA image recognition with 8.7x faster training
#4*11.5% as much compute.
Re: DeepMind achieves SOTA image recognition with 8.7x faster training
#5tl;dr: Don't use batch norm for preventing exploding gradients but adaptive gradient thresholds.
For this they compute the Frobenius norm (square root of the sum of squares) of the weight layer and its gradient and take the ratio of these as clipping threshold.
That saves the meta search for the optimal threshold but also is better than a fixed threshold could ever be.
Very simple idea.
Re: DeepMind achieves SOTA image recognition with 8.7x faster training
#6tl;dr: Don't use batch norm for preventing exploding gradients but adaptive gradient thresholds. For this they compute the Frobenius norm (square root of the sum of squares) of the weight layer and its gradient and take the ratio of these as clipping threshold. That saves the meta search for the optimal threshold but also is better than a fixed threshold could ever be. Very simple idea.
Thanks for macroexpanding frobnorm.
I'm skeptical that these hand-coded thresholds can ever match what a model can learn automatically. But it's hard to argue with results.
Re: DeepMind achieves SOTA image recognition with 8.7x faster training
#7tl;dr: Don't use batch norm for preventing exploding gradients but adaptive gradient thresholds. For this they compute the Frobenius norm (square root of the sum of squares) of the weight layer and its gradient and take the ratio of these as clipping threshold. That saves the meta search for the optimal threshold but also is better than a fixed threshold could ever be. Very simple idea.
Is this the first time that the gradient clip threshold has been chosen relative to the size of the weight matrix?
Re: DeepMind achieves SOTA image recognition with 8.7x faster training
#8The speed improvements are certainly interesting, the performance improvements seem decidedly not. This method has more than 2x the parameters of all but one of the models it was compared against.
If I’m off-base here can someone explain?
Re: DeepMind achieves SOTA image recognition with 8.7x faster training
#9Do they disclose any important techniques/ideas on how to achieve these results in the paper, or it's more of a technical press release?
Re: DeepMind achieves SOTA image recognition with 8.7x faster training
#10The big deal here is the removal of BatchNorm. People never really liked BatchNorm for various theoretical and practical reasons, and yet it was required for all the top performing models. If this allows us to get rid of it forever that will be really nice.