Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

DeepMind achieves SOTA image recognition with 8.7x faster training

arxiv.org

Post reply on HN
Print view View original

31–40 of 85 posts

Re: DeepMind achieves SOTA image recognition with 8.7x faster training

#31

Post by sillysaurusx »

gameshot911 wrote: post #24
sillysaurusx wrote: post #3

I was skeptical of the 83%-whatever top-1 accuracy on Imagenet. But someone pointed out that the accuracy increases when the model is pretrained on JFT, google's proprietary 100-million image dataset, the model's accuracy increases to 87%-whatever. That's pretty interesting. It implies that the original accuracy rating might be legit. The concern is that we're chasing the imagenet benchmark as if it's the holy grail,…

Those example pictures are trippy! Some of them look like those weird DeepMind sick-fever dream creations. Except I assume they are all real photos, not generated. It's very possible that a trained AI would be better-able to identify some of the candidates than I would. For example, from OP's post, w/ coordinate system starting at lower left, I have no idea what I'm looking at in these examples, except they look orga…

I'm surprised so many people want to see our BigGAN images. Thank you for asking :)

You can watch the training process here: http://song.tensorfork.com:8097/#images

It's been going on for a month and a half, but I leave it running mostly as a fishtank rather than to get to a specific objective. It's fun to load it up and look at a new random image whenever I want. Plus I like the idea of my little TPU being like "look at me! I'm doing work! Here's what I've prepared for you!" so I try to keep my little fella online all the time.

- https://i.imgur.com/0O5KZdE.png

- Plus stuff like this makes me laugh really hard. https://i.imgur.com/EnfIBz3.png

- Some nice flowers and a boat. https://i.imgur.com/mrFkIx0.png

The model is getting quite good. I kind of forgot about it over the past few weeks. StyleGAN could never get anywhere close to this level of detail. I had to spend roughly a year tracking down a crucial bug in the implementation that prevented biggan from working very well until now: https://github.com/google/compare_gan/issues/54

And we also seemed to solve BigGAN collapse, so theoretically the model can improve forever now. I leave it running to see how good it can get.

I've never seen conglomerate pictures like this used in AI training. Do you train models on these 4x4 images? What's the purpose vs a single picture at a time? Does the model know that you're feeding it 4x4 examples, or does it have to figure that out itself?

Nah, the grid is just for convenient viewing for humans. Robots see one image at a time. (Or more specifically, a batch of images; we happen to use batch size 2 or 4, I forget, so each core sees two images at a time, and then all 8 cores broadcast their gradients to each other and average, so it's really seeing 16 or 32 images at a time.)

I feel a bit silly plugging our community so much, but it's really true. If you like tricks like this, join the Tensorfork discord:

https://discord.com/invite/x52Xz3y

My theory when I set it up was that everyone has little tricks like this, but there's no central repository of knowledge / place to ask questions. But now that there are 1,200+ of us, it's become the de facto place to pop in and share random ideas and tricks.

For what it's worth, https://thisanimedoesnotexist.ai/ was a joint collaboration of several Tensorfork discord members. :)

If you want future updates about this specific BigGAN model, twitter is your best bet: https://twitter.com/search?q=(from%3Atheshawwn)%20biggan&src...

Re: DeepMind achieves SOTA image recognition with 8.7x faster training

#32

Post by sillysaurusx »

Jabbles wrote: post #30
sillysaurusx wrote: post #28

Earlier quoted context omitted.

Nope! It's an interesting balance. The truth of the situation seems to be: the generator and discriminator provide a "signal" to each other, like two planets orbiting around each other. If you cut the signal from one, the other will rapidly veer off into infinity, i.e. collapse quickly. Or it will veer off in the other direction, i.e. all progress will stop and the model won't improve. So it's a constant "signal", yo…

Thanks for the detailed answer. I don't claim it to be a novel idea, I just remember the Alpha Go (zero?) paper that said they played it against older versions to make sure it hadn't got into a bad state.

Ah! This is an interesting difference, and illustrates one fun aspect of GANs vs other types of models: Alpha Go had a very specific "win condition" that you can measure precisely. (Can the model win the game?)

Whereas it's very difficult to quantify what it means to be "better" at generating images, once you get to a certain threshold of realism. (Was Leonardo better than michelangelo? Probably, but it's hard to measure precisely.)

The way that Alpha Go worked was, it gathered a bunch of experiences, i.e. it played a bunch of games. Then, after playing tons of games -- tens of dozens! just kidding, probably like 20 million -- it then performed a single gradient update.

In other words, you gather your current experiences, and then you react to them. It's a two-phase commit. There's an explicit "gather" step, which you then react to by updating your database of parameters, so to speak.

Whereas with GANs, that happens continuously. There's no "gather" step. The generator simply tries to maximize the discrimiantor's loss, and the discriminator tries to minimize it.

Balancing the two has been very tricky. But the results speak for themselves.

Re: DeepMind achieves SOTA image recognition with 8.7x faster training

#33

Post by 6gvONxR4sf7o »

modeless wrote: post #11
smeeth wrote: post #8

The speed improvements are certainly interesting, the performance improvements seem decidedly not. This method has more than 2x the parameters of all but one of the models it was compared against. If I’m off-base here can someone explain?

I don't care how many parameters my model has per se. What I care about is how expensive it is to train in time and dollars. If this makes it cheaper to train better models despite more parameters, that's still a win.

Some models are still memory limited. Fewer parameters are very useful in those settings.

Re: DeepMind achieves SOTA image recognition with 8.7x faster training

#34

Post by Itsdijital »

sillysaurusx wrote: post #31
gameshot911 wrote: post #24

Earlier quoted context omitted.

Those example pictures are trippy! Some of them look like those weird DeepMind sick-fever dream creations. Except I assume they are all real photos, not generated. It's very possible that a trained AI would be better-able to identify some of the candidates than I would. For example, from OP's post, w/ coordinate system starting at lower left, I have no idea what I'm looking at in these examples, except they look orga…

I'm surprised so many people want to see our BigGAN images. Thank you for asking :) You can watch the training process here: http://song.tensorfork.com:8097/#images It's been going on for a month and a half, but I leave it running mostly as a fishtank rather than to get to a specific objective. It's fun to load it up and look at a new random image whenever I want. Plus I like the idea of my little TPU being like "loo…

This is awesome, thanks.

Re: DeepMind achieves SOTA image recognition with 8.7x faster training

#35

Post by throwaway189262 »

I have a feeling the ML community is going to pivot focus to faster and smaller training before larger advancements are made. It's simply too expensive for much AI research to happen when state of the art models take 500k of hardware to train.

For all the mathematician hype around ML research, much of the work is closer to alchemy than science. We simply don't understand a great deal of why these neural nets work.

The people doing math above algebra are few and the scene is dominated by "guess and check" style model tinkering.

Many "state of the art models" are simply a bunch of common strategies glued together in a way researchers found worked the best (by trying a bunch of different ones).

An average Joe could probably write influential ML papers by gluing RNN/GAN layers to existing models and fiddling with the parameters until they beat current state of the art. In fact, in NLP models, this is essentially what has happened with roBERTa, XLNET, ELECTRA, etc. They're all somewhat trivial variations on Google's BERT, which is more creative but yet again built on existing models.

Anyways, my point is, none of this required math or genius or particularly demanding thought. It was basically let's tinker with this until we find a way that's better, using guess and check. No equations needed.

We are a long way from the type of simulations done for protein folding and materials strength and basically every other scientific field. It's still the wild west

Re: DeepMind achieves SOTA image recognition with 8.7x faster training

#36

Post by dontreact »

modeless wrote: post #11
smeeth wrote: post #8

The speed improvements are certainly interesting, the performance improvements seem decidedly not. This method has more than 2x the parameters of all but one of the models it was compared against. If I’m off-base here can someone explain?

I don't care how many parameters my model has per se. What I care about is how expensive it is to train in time and dollars. If this makes it cheaper to train better models despite more parameters, that's still a win.

In cases where you have to deploy the model and you are limited in terms of flops, this paper does not help much, unless it’s removal of batchnorm somehow allows a future network that is actually faster at inference time.

Re: DeepMind achieves SOTA image recognition with 8.7x faster training

#37

Post by The_rationalist »

This isn't the real SOTA, "Meta pseudo labels" has ~10% less errors, while having less parameters. https://paperswithcode.com/sota/image-classification-on-imag... However the fast training is an interesting property.

It would be interesting to test thoses efficientNets with zeroth order backpropagation as it allows a 300X speedup (vs 8.7x) while not regressing accuracy that much https://paperswithcode.com/paper/zorb-a-derivative-free-back...

Re: DeepMind achieves SOTA image recognition with 8.7x faster training

#38

Post by quantumofalpha »

Jabbles wrote: post #18
sillysaurusx wrote: post #3

I was skeptical of the 83%-whatever top-1 accuracy on Imagenet. But someone pointed out that the accuracy increases when the model is pretrained on JFT, google's proprietary 100-million image dataset, the model's accuracy increases to 87%-whatever. That's pretty interesting. It implies that the original accuracy rating might be legit. The concern is that we're chasing the imagenet benchmark as if it's the holy grail,…

What safeguards are there or what assurances do we have that JFT is not contaminated with images from (or extremely similar to) the validation set?

Just the sheer size of JFT (latest versions I heard approach 1B images), so it's probably impractical to train on it till overfitting.

Re: DeepMind achieves SOTA image recognition with 8.7x faster training

#40

Post by modeless »

dontreact wrote: post #36
modeless wrote: post #11

Earlier quoted context omitted.

I don't care how many parameters my model has per se. What I care about is how expensive it is to train in time and dollars. If this makes it cheaper to train better models despite more parameters, that's still a win.

In cases where you have to deploy the model and you are limited in terms of flops, this paper does not help much, unless it’s removal of batchnorm somehow allows a future network that is actually faster at inference time.

There are a lot of techniques for sparsifying or pruning or distilling models to reduce inference FLOPS, and they almost always produce better results when starting with a better model. Also, if your model is 8x faster to train at the same size then you can do 8x as much hyperparameter tuning and get a better result.
Post reply on HN