Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

DeepMind achieves SOTA image recognition with 8.7x faster training

arxiv.org

Post reply on HN
Print view View original

41–50 of 85 posts

Re: DeepMind achieves SOTA image recognition with 8.7x faster training

#41

Post by david-gpu »

dontreact wrote: post #36
modeless wrote: post #11

Earlier quoted context omitted.

I don't care how many parameters my model has per se. What I care about is how expensive it is to train in time and dollars. If this makes it cheaper to train better models despite more parameters, that's still a win.

In cases where you have to deploy the model and you are limited in terms of flops, this paper does not help much, unless it’s removal of batchnorm somehow allows a future network that is actually faster at inference time.

But for deployment in smaller devices you can use techniques such as distillation, quantization and sparsity. Training and inference are very different problems in practice.

Re: DeepMind achieves SOTA image recognition with 8.7x faster training

#42

Post by The_rationalist »

throwaway189262 wrote: post #35

I have a feeling the ML community is going to pivot focus to faster and smaller training before larger advancements are made. It's simply too expensive for much AI research to happen when state of the art models take 500k of hardware to train. For all the mathematician hype around ML research, much of the work is closer to alchemy than science. We simply don't understand a great deal of why these neural nets work. Th…

While I generally agree to some extent: roBERTa, XLNET, ELECTRA, etc. They're all somewhat trivial variations on Google's BERT, which is more creative Researchers take inspirations from existing models of course and some BERT derivatives are trivial. However, XLnet is in it's own league, while the author (a genius chinese student) was inspired by BERT it is one of the few SOTA pré trained models to be not based on BERT and is actually an auto regressive one! Such difference allow it to be better at many things as it doesn't has to corrupt the tokens (from my shallow understanding). This model is two years old but is still sadly the one that ranks the most SOTA in key tasks e.g dependency parsing. And after all those time nobody cared enough to even test it on other foundational tasks (which is extremely sad and pathetic) like e.g coreference resolution. Sadly because of conformism effects almost zero researcher has created XLnet derivatives. Almost all researchers continue to search in the local minima that is BERT, which I find, immensely ironic.

While ad hoc empirical fine tuning is a big part of improving sota, mathematical genius can still enable revolutions e.g this recent alternative to classical backpropagation that is 300X faster with low accuracy loss https://paperswithcode.com/paper/zorb-a-derivative-free-back...

Re: DeepMind achieves SOTA image recognition with 8.7x faster training

#43

Post by throwaway189262 »

The_rationalist wrote: post #42
throwaway189262 wrote: post #35

I have a feeling the ML community is going to pivot focus to faster and smaller training before larger advancements are made. It's simply too expensive for much AI research to happen when state of the art models take 500k of hardware to train. For all the mathematician hype around ML research, much of the work is closer to alchemy than science. We simply don't understand a great deal of why these neural nets work. Th…

While I generally agree to some extent: roBERTa, XLNET, ELECTRA, etc. They're all somewhat trivial variations on Google's BERT, which is more creative Researchers take inspirations from existing models of course and some BERT derivatives are trivial. However, XLnet is in it's own league, while the author (a genius chinese student) was inspired by BERT it is one of the few SOTA pré trained models to be not based on BE…

Interesting, but in not sure you're completely right about XLNET. I heard it takes an absurd amount of resources to train. Even more than the BERT variations. And this is likely why there's not a ton of interest in it

Re: DeepMind achieves SOTA image recognition with 8.7x faster training

#44

Post by The_rationalist »

trott wrote: post #16
dheera wrote: post #2

> 8.7x faster to train This is an achievement but it would be helpful to have put "to train" in the title as this is quite different from efficiency at inference time, which is what often actually matters in deployed applications. From Table 3 on Page 7 it appears to me that NFNet is significantly heavier in the number of parameters than EfficientNet for similar accuracies. For example EffNet-B5 achieves 83.7% with 3…

> It appears to me at first glance that NFNet has not achieved SOTA at inference. It has, for larger models (F1 vs B7). See Fig 4 in the Appendix.

No it hasn't https://paperswithcode.com/sota/image-classification-on-imag...

Re: DeepMind achieves SOTA image recognition with 8.7x faster training

#45

Post by belgian_guy »

nullifidian wrote: post #9

Do they disclose any important techniques/ideas on how to achieve these results in the paper, or it's more of a technical press release?

Yes they do (arxiv is a place for scientific papers not press releases). I've only skimmed it, but the paper introduce an adaptive way to clip gradients. Meaning that if the ratio of the gradient norm to weight norm surpasses a certain threshold, they clip it. This stabilizes learning and seems to avoid the need for batch normalization. Seems quite promising imo and something that could stick (I'm quite happy if we could finally do away with batchnorm).

Re: DeepMind achieves SOTA image recognition with 8.7x faster training

#46

Post by code-scope »

sillysaurusx wrote: post #3

I was skeptical of the 83%-whatever top-1 accuracy on Imagenet. But someone pointed out that the accuracy increases when the model is pretrained on JFT, google's proprietary 100-million image dataset, the model's accuracy increases to 87%-whatever. That's pretty interesting. It implies that the original accuracy rating might be legit. The concern is that we're chasing the imagenet benchmark as if it's the holy grail,…

Is there any (influence) SW framework that takes youtube video as input and split out object/timestamp as output?

Re: DeepMind achieves SOTA image recognition with 8.7x faster training

#47

Post by trott »

The_rationalist wrote: post #44
trott wrote: post #16

Earlier quoted context omitted.

> It appears to me at first glance that NFNet has not achieved SOTA at inference. It has, for larger models (F1 vs B7). See Fig 4 in the Appendix.

No it hasn't https://paperswithcode.com/sota/image-classification-on-imag...

> No it hasn't https://paperswithcode.com/sota/image-classification-on-imag...

We were talking about models trained on ImageNet, specifically about the trade-off between accuracy and FLOPs. But the higher-accuracy models listed in your link use extra data. So it's not quite the same benchmark we were talking about.

Re: DeepMind achieves SOTA image recognition with 8.7x faster training

#48

Post by sillysaurusx »

throwaway189262 wrote: post #35

I have a feeling the ML community is going to pivot focus to faster and smaller training before larger advancements are made. It's simply too expensive for much AI research to happen when state of the art models take 500k of hardware to train. For all the mathematician hype around ML research, much of the work is closer to alchemy than science. We simply don't understand a great deal of why these neural nets work. Th…

Apply to TFRC! https://www.tensorflow.org/tfrc

They are very permissive. And you get to play with $500k worth of hardware. Been a member for over a year now. Jonathan is singlehandedly the best support person I've ever worked with, or perhaps ever will work with.

I would've completely agreed with you if not for TFRC. And I couldn't resist the opportunity of playing with some big metal, even if it's hard to work with.

Re: DeepMind achieves SOTA image recognition with 8.7x faster training

#49

Post by trott »

highfrequency wrote: post #14

They compare the training latency for different models with a fixed batch size of 32. But if the DeepMind models are several times larger than the comparison models in each latency class, it seems that the comparison models could use larger batch sizes for faster overall training time.

For ConvNets, the memory use of the models themselves is pretty modest. For example, even with 0.5B parameters, with FP32, weights+gradients+momentum should use just 6GB (unless your framework sucks, or you have extra overhead from distributed training) So, if your model is twice smaller, you'll only save 3GB. If your VRAM is 32GB, saving 3GB won't let you use a much bigger batch size. On the other hand, the absence of batch norm can actually lead to memory savings proportional to batch size.

Re: DeepMind achieves SOTA image recognition with 8.7x faster training

#50

Post by dontreact »

david-gpu wrote: post #41
dontreact wrote: post #36

Earlier quoted context omitted.

In cases where you have to deploy the model and you are limited in terms of flops, this paper does not help much, unless it’s removal of batchnorm somehow allows a future network that is actually faster at inference time.

But for deployment in smaller devices you can use techniques such as distillation, quantization and sparsity. Training and inference are very different problems in practice.

Yes but you can do that with efficientnet as well. The point is that this is an improvement only for training because it uses computations which are highly optimized on TPU
Post reply on HN