Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

DeepMind achieves SOTA image recognition with 8.7x faster training

arxiv.org

Post reply on HN
Print view View original

81–85 of 85 posts

Re: DeepMind achieves SOTA image recognition with 8.7x faster training

#81

Post by woko »

throwaway189262 wrote: post #35

I have a feeling the ML community is going to pivot focus to faster and smaller training before larger advancements are made. It's simply too expensive for much AI research to happen when state of the art models take 500k of hardware to train. For all the mathematician hype around ML research, much of the work is closer to alchemy than science. We simply don't understand a great deal of why these neural nets work. Th…

I see your point when it comes to paper publication in general, but I feel that your post is very unwarranted with respect to the original post: DeepMind has recruited top-quality theoretical researchers from public institutions, it is not just experimental work which your average Joe could do with a few guesses. These researchers published a lot of theoretical papers before they were recruited, and they still publish a lot of them now that they are working at DeepMind, but they have more computational hardware to apply their ideas.

Here is just one uncurated example of a publication: - https://deepmind.com/research/publications/Taylor-Expansion-... - https://arxiv.org/pdf/2003.06259.pdf

Re: DeepMind achieves SOTA image recognition with 8.7x faster training

#82

Post by physicsguy »

throwaway189262 wrote: post #35

I have a feeling the ML community is going to pivot focus to faster and smaller training before larger advancements are made. It's simply too expensive for much AI research to happen when state of the art models take 500k of hardware to train. For all the mathematician hype around ML research, much of the work is closer to alchemy than science. We simply don't understand a great deal of why these neural nets work. Th…

Minor variations on top of existing stuff and occasional leaps forward is most research though, it's not surprising ML research follows a similar pattern.

Re: DeepMind achieves SOTA image recognition with 8.7x faster training

#83

Post by The_rationalist »

Der_Einzige wrote: post #78
The_rationalist wrote: post #42

Earlier quoted context omitted.

While I generally agree to some extent: roBERTa, XLNET, ELECTRA, etc. They're all somewhat trivial variations on Google's BERT, which is more creative Researchers take inspirations from existing models of course and some BERT derivatives are trivial. However, XLnet is in it's own league, while the author (a genius chinese student) was inspired by BERT it is one of the few SOTA pré trained models to be not based on BE…

Not sure why you're being downvoted. I was about to swoop in and mention that the top level comment was wrong about XLNet being some bert based model but you beat me to it. Sometimes HN is full of people who think they know what they're talking about but just don't. This is one of those times.

Honestly my comments that have negative Karma have generally much more useful truths than my comments that have positive Karma, this is almost systemic. This show how low quality the HN community is, epistemologically speaking. There are much less lazy communities out there like lesswrong.com but unfortunately they don't talk much about computer science

Re: DeepMind achieves SOTA image recognition with 8.7x faster training

#85

Post by trott »

nl wrote: post #61
The_rationalist wrote: post #39

Earlier quoted context omitted.

But what was the baseline hardware for a reasonable training time?

Page 7 has a table of one training step on TPUv3 and V100 GPUs. I don't completely understand this: NFNet is slower than its competitors on this benchmark, but they claim higher efficiency. This isn't obvious to me.

> I don't completely understand this: NFNet is slower than its competitors on this benchmark, but they claim higher efficiency.

Take a look at F1 and B7: They have the same accuracy, but F1 is smaller and much faster.

Post reply on HN