Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

DeepMind achieves SOTA image recognition with 8.7x faster training

arxiv.org

Post reply on HN
Print view View original

71–80 of 85 posts

Re: DeepMind achieves SOTA image recognition with 8.7x faster training

#71

Post by rsfern »

eggie5 wrote: post #63

Do what do attribute to the gains? Adaptive clipping? Or $$$ spent on NAS??

They do a little study in section 4.1 comparing batchnorm to adaptive gradient clipping for resnets over a range of hyperparameters, and they also compare perf to batchnorm versions in table 6. The results indicate AGC does give a real boost over batchnorm

They do a bunch of manual hyperparameter tuning that seems necessary to get the state of the art results, from my reading it doesn’t seem like they actually used NAS. Just that the baseline they compare to was found with NAS

Re: DeepMind achieves SOTA image recognition with 8.7x faster training

#72

Post by rsfern »

eggie5 wrote: post #64
belgian_guy wrote: post #45

Earlier quoted context omitted.

Yes they do (arxiv is a place for scientific papers not press releases). I've only skimmed it, but the paper introduce an adaptive way to clip gradients. Meaning that if the ratio of the gradient norm to weight norm surpasses a certain threshold, they clip it. This stabilizes learning and seems to avoid the need for batch normalization. Seems quite promising imo and something that could stick (I'm quite happy if we c…

You missed a big part: they did a big NAS run to make it work.

Where did you see that they used NAS? Their preliminary results show it works even for the baseline model

they did a lot of manual hyperparameter optimization, and spend a fair amount of time unpacking the rationale for their choices, including a negative results section (!) in the appendix

Re: DeepMind achieves SOTA image recognition with 8.7x faster training

#73

Post by throwaway189262 »

tbalsam wrote: post #54
throwaway189262 wrote: post #35

I have a feeling the ML community is going to pivot focus to faster and smaller training before larger advancements are made. It's simply too expensive for much AI research to happen when state of the art models take 500k of hardware to train. For all the mathematician hype around ML research, much of the work is closer to alchemy than science. We simply don't understand a great deal of why these neural nets work. Th…

This feels much like the sentiment in the field about two years ago or so. While I feel like the "alchemy" storyline is still somewhat in play, most of the big important parts of the deep learning process have enough ideological linear approximators stacked around them that if you know what you're doing or looking at, you can jump to an unexplored trench with some reasonable feeling about whether you'll get something…

Good write-up. Indeed I'm a novice tinkering with a decent gaming GPU :) . I was initially daunted by ML but the more I read I began to realize the field is quite accessible these days. Most of the time, you don't need to understand why or how this stuff works at a deep level. You just need a good feel of what might work and a training dataset.

Much of that is the enormous amount of work done plastering over complex GPU programming. But some of it is the tinkering nature of solving ML problems.

The field I'm most interested in right now for instance, NLP, is highly dataset dependent. It's fairly easy to exceed SoTA right now using open sourced models if you have a better, more specialized dataset than what's freely available.

Re: DeepMind achieves SOTA image recognition with 8.7x faster training

#75

Post by eachro »

throwaway189262 wrote: post #35

I have a feeling the ML community is going to pivot focus to faster and smaller training before larger advancements are made. It's simply too expensive for much AI research to happen when state of the art models take 500k of hardware to train. For all the mathematician hype around ML research, much of the work is closer to alchemy than science. We simply don't understand a great deal of why these neural nets work. Th…

> An average Joe could probably write influential ML papers by gluing RNN/GAN layers to existing models and fiddling with the parameters until they beat current state of the art.

Right, but you have to remember there are legions of grad students doing exactly this so it ends up being quite competitive to churn out papers this way.

Re: DeepMind achieves SOTA image recognition with 8.7x faster training

#76

Post by Yenrabbit »

Yannic Kilcher has a great video on this out already, including some thoughtful critique. https://youtu.be/rNkHjZtH0RQ

One of the things he mentioned was that they introduce this fancy alternative to BatchNorm AND come up with a fancy new architecture. The combination does really well, but it isn't clear how much of the gain is due to the new improved architecture Vs the 'adaptive gradient clipping' they introduce.

Re: DeepMind achieves SOTA image recognition with 8.7x faster training

#77

Post by visarga »

throwaway189262 wrote: post #35

I have a feeling the ML community is going to pivot focus to faster and smaller training before larger advancements are made. It's simply too expensive for much AI research to happen when state of the art models take 500k of hardware to train. For all the mathematician hype around ML research, much of the work is closer to alchemy than science. We simply don't understand a great deal of why these neural nets work. Th…

> Anyways, my point is, none of this required math or genius or particularly demanding thought. It was basically let's tinker with this until we find a way that's better, using guess and check. No equations needed.

I get that you'd like to have a clear theoretical basis for what works and we're far from there. But in the meantime we're stumbling in the dark, discovering tricks and forming intuitions, not knowing even where the road is going to lead us.

This is an evolutionary process of ideas, similar to biological evolution that managed to make us. If you know where you're going you can optimise your actions but when you don't even know what might be useful later on, then all attempts are good. They increase diversity and discover blind spots. Some of them will be the stepping stones for the future, but we can't say in advance which and how.

Link to a long discussion about the evolution of ML ideas and the book "Why greatness cannot be planned" by Kenneth Stanley - https://youtu.be/lhYGXYeMq_E?t=416

Re: DeepMind achieves SOTA image recognition with 8.7x faster training

#78

Post by Der_Einzige »

The_rationalist wrote: post #42
throwaway189262 wrote: post #35

I have a feeling the ML community is going to pivot focus to faster and smaller training before larger advancements are made. It's simply too expensive for much AI research to happen when state of the art models take 500k of hardware to train. For all the mathematician hype around ML research, much of the work is closer to alchemy than science. We simply don't understand a great deal of why these neural nets work. Th…

While I generally agree to some extent: roBERTa, XLNET, ELECTRA, etc. They're all somewhat trivial variations on Google's BERT, which is more creative Researchers take inspirations from existing models of course and some BERT derivatives are trivial. However, XLnet is in it's own league, while the author (a genius chinese student) was inspired by BERT it is one of the few SOTA pré trained models to be not based on BE…

Not sure why you're being downvoted. I was about to swoop in and mention that the top level comment was wrong about XLNet being some bert based model but you beat me to it.

Sometimes HN is full of people who think they know what they're talking about but just don't. This is one of those times.

Re: DeepMind achieves SOTA image recognition with 8.7x faster training

#80

Post by tbalsam »

throwaway189262 wrote: post #73
tbalsam wrote: post #54

Earlier quoted context omitted.

This feels much like the sentiment in the field about two years ago or so. While I feel like the "alchemy" storyline is still somewhat in play, most of the big important parts of the deep learning process have enough ideological linear approximators stacked around them that if you know what you're doing or looking at, you can jump to an unexplored trench with some reasonable feeling about whether you'll get something…

Good write-up. Indeed I'm a novice tinkering with a decent gaming GPU :) . I was initially daunted by ML but the more I read I began to realize the field is quite accessible these days. Most of the time, you don't need to understand why or how this stuff works at a deep level. You just need a good feel of what might work and a training dataset. Much of that is the enormous amount of work done plastering over complex…

Absolutely, couldn't agree more. If you want a secret, just find what scaling laws are there and find tunnels to bypass them. There's always a way to the secret garden, you just sometimes have to look long and hard... ;)

I started with my 1070 flat, and have had some people far, far, far smarter and more experienced than me help me understand a lot of the underlying mathematics a lot. Semi-supervised/bootstrapping may be a fun topic, if you can avoid the giant CAT trucks of the FAANG monoliths blazing through there, and there's always really good artisanal work to be done if you can prove certain mathematical conditions hold such that other (oftentimes counterintuitive and bizzare) operators still work, or work when they shouldn't before.

You could also get into the rat race of the *formers -- the Nyströmformer is quite spectacular and nearly linear, and yes, if you're hot on your feet and clever enough, you might be able to beat everything into submission.

Also, distrust every non-bayesian thing involving means and sigmas. Those are always ad hoc and beat the real data manifold into submission, which really does a disservice a lot of the time, I think. There's a lot to get around that (I suppose including the above, which I'd forgotten about, but there's always, uh, SeLU if you're looking for inspiration plus a phenomenal appendix. You want universal attractors? Set up and prove something that's more amenable to a good manifold structure than simply a certain distribution of activations -- that truly tells us nothing!)

Hope those are fun ideas -- and my deepest apologies if I was uncharitable to you in my former post. I went back and edited it for politeness but reading it again felt some of my earlier aggression fall through, and I'm certainly sorry about that -- I should be helping new folks, not being an aggressive gatekeeper against that.

In any case, so long as you're able to keep mathematical interest, there's always a nice hole to square yourself away into. Talk to a good accomplished research professional and they might be able to point you in fun directions (aside from my personal noobishness ;))

Let me know if any of those catch your eye and end up going anywhere, I'm happy to help when it moves the field forward! :)))

Post reply on HN