61–70 of 85 posts
*11.5% as much compute.
But what was the baseline hardware for a reasonable training time?
I don't completely understand this: NFNet is slower than its competitors on this benchmark, but they claim higher efficiency. This isn't obvious to me.
Earlier quoted context omitted.
> No it hasn't https://paperswithcode.com/sota/image-classification-on-imag ... We were talking about models trained on ImageNet, specifically about the trade-off between accuracy and FLOPs. But the higher-accuracy models listed in your link use extra data. So it's not quite the same benchmark we were talking about.
The deepmind paper NFNet-F4+ you were talking about also has external training data. The number one in accuracy (Meta pseudo labels) is also faster for inference (390M vs 570M parameters) vs the deepmind one. So what are you disagreeing with?
@dheera and I did not mention NFNet-F4+. All models, tables, figures and numbers that we did mention resulted from training on ImageNet alone.
Do they disclose any important techniques/ideas on how to achieve these results in the paper, or it's more of a technical press release?
Yes they do (arxiv is a place for scientific papers not press releases). I've only skimmed it, but the paper introduce an adaptive way to clip gradients. Meaning that if the ratio of the gradient norm to weight norm surpasses a certain threshold, they clip it. This stabilizes learning and seems to avoid the need for batch normalization. Seems quite promising imo and something that could stick (I'm quite happy if we c…
I have a feeling the ML community is going to pivot focus to faster and smaller training before larger advancements are made. It's simply too expensive for much AI research to happen when state of the art models take 500k of hardware to train. For all the mathematician hype around ML research, much of the work is closer to alchemy than science. We simply don't understand a great deal of why these neural nets work. Th…
Have you not heard of alphafold?
Earlier quoted context omitted.
There are a lot of techniques for sparsifying or pruning or distilling models to reduce inference FLOPS, and they almost always produce better results when starting with a better model. Also, if your model is 8x faster to train at the same size then you can do 8x as much hyperparameter tuning and get a better result.
This model is much more expensive than efficientnet at inference (I think the flops are about 2x?). You can use these same techniques with efficientnet.
I have a feeling the ML community is going to pivot focus to faster and smaller training before larger advancements are made. It's simply too expensive for much AI research to happen when state of the art models take 500k of hardware to train. For all the mathematician hype around ML research, much of the work is closer to alchemy than science. We simply don't understand a great deal of why these neural nets work. Th…
There's a lot of interest in various ML communities on more efficient training and inference. Both vision and NLP have had a growing focus on these problems in recent years. I think you make a good observation that much of ML progress is driven by tinkering with existing models, though instead of describing it as more "alchemy than science" it's probably more accurate to say it's very experimental right now. Being ve…
I have a feeling the ML community is going to pivot focus to faster and smaller training before larger advancements are made. It's simply too expensive for much AI research to happen when state of the art models take 500k of hardware to train. For all the mathematician hype around ML research, much of the work is closer to alchemy than science. We simply don't understand a great deal of why these neural nets work. Th…
>We are a long way from the type of simulations done for protein folding and materials strength and basically every other scientific field. Have you not heard of alphafold?
https://deepmind.com/blog/article/alphafold-a-solution-to-a-...
Earlier quoted context omitted.
This model is much more expensive than efficientnet at inference (I think the flops are about 2x?). You can use these same techniques with efficientnet.
> they almost always produce better results when starting with a better model.