eli5 what SOTA image recognition is?
State of The Art == SOTA
You might enjoy paperswithcode.com
DeepMind achieves SOTA image recognition with 8.7x faster training
21–30 of 85 posts
Re: DeepMind achieves SOTA image recognition with 8.7x faster training
#22I was skeptical of the 83%-whatever top-1 accuracy on Imagenet. But someone pointed out that the accuracy increases when the model is pretrained on JFT, google's proprietary 100-million image dataset, the model's accuracy increases to 87%-whatever. That's pretty interesting. It implies that the original accuracy rating might be legit. The concern is that we're chasing the imagenet benchmark as if it's the holy grail,…
Where are these images from? Are there more?
Oh, you! I'm so flattered. You're making me blush.
Sure, you can have as many as you want. Watch it train in real time:
http://song.tensorfork.com:8097/#images
Though I imagine HN might swamp our little training server running tensorboard, so here you go.
https://i.imgur.com/BkGxbo7.png
We've been training a BigGAN-Deep model for 1.5 months now. Though that sounds like a long time, in reality it's completely automatic and I've been leaving it running just to see what will happen. Every single other BigGAN implementation reports that eventually the training run will collapse. We observed the same thing. But gwern came up with a brilliantly simple way to solve this:
if D_loss If you like this sort of thing in general, I encourage you to come join the Tensorfork community discord server, which we affectionately named the "TPU podcast": https://discord.com/invite/x52Xz3y
There are some 1,200 of us now, and people are always showing off stuff like this in our (way too many) channels.
Re: DeepMind achieves SOTA image recognition with 8.7x faster training
#23eli5 what SOTA image recognition is?
SOTA is 'state of the art'. Image recognition is a task classically appraised by calculating the accuracy on the ImageNet dataset, which requires a system to classify images each as one of 1,000 pre-determined classes.
so how many images does the current SOTA take to train a classifier? Trying to gauge how much of an improvement Deepmind has made here.
Re: DeepMind achieves SOTA image recognition with 8.7x faster training
#24I was skeptical of the 83%-whatever top-1 accuracy on Imagenet. But someone pointed out that the accuracy increases when the model is pretrained on JFT, google's proprietary 100-million image dataset, the model's accuracy increases to 87%-whatever. That's pretty interesting. It implies that the original accuracy rating might be legit. The concern is that we're chasing the imagenet benchmark as if it's the holy grail,…
Those example pictures are trippy! Some of them look like those weird DeepMind sick-fever dream creations. Except I assume they are all real photos, not generated. It's very possible that a trained AI would be better-able to identify some of the candidates than I would.
For example, from OP's post, w/ coordinate system starting at lower left, I have no idea what I'm looking at in these examples, except they look organic-ish: [1]: [1,4], [3,2], [4,1]
sillysaurusx: I've never seen conglomerate pictures like this used in AI training. Do you train models on these 4x4 images? What's the purpose vs a single picture at a time? Does the model know that you're feeding it 4x4 examples, or does it have to figure that out itself?
Aside: Another awesome 'sick-fever dream creation' example if you missed it when it made the rounds on HN is this[3]. Slide the creativity filter up for weirdness!
Re: DeepMind achieves SOTA image recognition with 8.7x faster training
#25Earlier quoted context omitted.
Where are these images from? Are there more?
Oh, you! I'm so flattered. You're making me blush. Sure, you can have as many as you want. Watch it train in real time: http://song.tensorfork.com:8097/#images Though I imagine HN might swamp our little training server running tensorboard, so here you go. https://i.imgur.com/BkGxbo7.png We've been training a BigGAN-Deep model for 1.5 months now. Though that sounds like a long time, in reality it's completely automati…
Do you take advantage of previous iterations of the generator and discriminator? i.e. the generator should be able to fool all previous discriminators, and the discriminator should be able to recognise the work of all previous generators?
Re: DeepMind achieves SOTA image recognition with 8.7x faster training
#26Earlier quoted context omitted.
SOTA is 'state of the art'. Image recognition is a task classically appraised by calculating the accuracy on the ImageNet dataset, which requires a system to classify images each as one of 1,000 pre-determined classes.
so how many images does the current SOTA take to train a classifier? Trying to gauge how much of an improvement Deepmind has made here.
The results that should be used to compare their results to others involves training on just under 1.3 million images across the 1,000 classes.
Their best results involve 'pretraining' on a dataset of 300 million examples, before 'tuning' it on the actual ImageNet training dataset as above.
Re: DeepMind achieves SOTA image recognition with 8.7x faster training
#27I was skeptical of the 83%-whatever top-1 accuracy on Imagenet. But someone pointed out that the accuracy increases when the model is pretrained on JFT, google's proprietary 100-million image dataset, the model's accuracy increases to 87%-whatever. That's pretty interesting. It implies that the original accuracy rating might be legit. The concern is that we're chasing the imagenet benchmark as if it's the holy grail,…
What safeguards are there or what assurances do we have that JFT is not contaminated with images from (or extremely similar to) the validation set?
Haha. None whatsoever.
The assurance is that everyone in the field seems to take the work seriously. But the reality is that errors creep in from a variety of corners. I would not be even slightly surprised to find that the validation data is substantially similar. We're still at the "bangs rocks together to make fire" phase of ML, which is both exciting and challenging; we're building the future from the ground up.
People rarely take the time to look at the actual images, but if you do, you'll notice they have some interesting errors in them: https://twitter.com/theshawwn/status/1262535747975868418
I built an interactive viewer for the tagging site: https://tags.tagpls.com/
(Someone tagged all 70 shoes in this one, which was kind of impressive... https://tags.shawwn.com/tags/https://battle.shawwn.com/sdc/i... )
Anyway, some of the validation images happen to be rotated 90 degrees and no one noticed. That made me wonder what other sorts of unexpected errors are in these specific 50,000 validation images that the world just-so-happened to decide were Super Important to the future of AI.
The trouble is, images in general are substantially similar to the imagenet validation dataset. In other words, it's tempting to try to think of some way of "dividing up" the data so that there's some sort of validation phase that you can cleanly separate. But reality isn't so kind. When you're at the scale of millions of images, holding out 10% is just a way of sanity checking that your model isn't memorizing the training data; nothing more.
Besides, random 90 degree rotations are introduced on purpose now, so it's funny that old mistakes tend not to matter.
Re: DeepMind achieves SOTA image recognition with 8.7x faster training
#28Earlier quoted context omitted.
Oh, you! I'm so flattered. You're making me blush. Sure, you can have as many as you want. Watch it train in real time: http://song.tensorfork.com:8097/#images Though I imagine HN might swamp our little training server running tensorboard, so here you go. https://i.imgur.com/BkGxbo7.png We've been training a BigGAN-Deep model for 1.5 months now. Though that sounds like a long time, in reality it's completely automati…
Do you take advantage of previous iterations of the generator and discriminator? i.e. the generator should be able to fool all previous discriminators, and the discriminator should be able to recognise the work of all previous generators?
Nope! It's an interesting balance. The truth of the situation seems to be: the generator and discriminator provide a "signal" to each other, like two planets orbiting around each other.
If you cut the signal from one, the other will rapidly veer off into infinity, i.e. collapse quickly. Or it will veer off in the other direction, i.e. all progress will stop and the model won't improve.
So it's a constant "signal", you see, where one is dependent on the other in the current state. Therefore I am skeptical of attempts to use previous states of discriminators.
However! One of the counterintuitive aspects of AI is that the strangest-sounding ideas often have a chance of being good ideas. It's also so hard to try new ideas that you have to pick specific ones. So, roll up your sleeves and implement yours; I would personally be delighted to see what the code would look like for "the current generator can fool all previous discriminators".
I really do not mean that in any sort of negative or dismissive way. I really hope that you will come try it, because DL has never been more accessible. And the time is ripe for fresh takes on old ideas; there's a very real chance that you'll stumble across something that works quite well, if you follow your line of thinking.
But for practical purposes, the current theory with generators and discriminators is that they react to their current states. So there's not really any way of testing "can the generator fool all previous discriminators?" because in reality, the generator isn't fooling the discriminator at all -- they simply notice when each other deviates by a small amount, and they make a corresponding "small delta change" in response. Kind of like an ongoing chess game.
Re: DeepMind achieves SOTA image recognition with 8.7x faster training
#29I was skeptical of the 83%-whatever top-1 accuracy on Imagenet. But someone pointed out that the accuracy increases when the model is pretrained on JFT, google's proprietary 100-million image dataset, the model's accuracy increases to 87%-whatever. That's pretty interesting. It implies that the original accuracy rating might be legit. The concern is that we're chasing the imagenet benchmark as if it's the holy grail,…
Not sure if I follow your JFT argument, but there's a large body of work on both (a) studying whether chasing ImageNet accuracy yields models that generalize well to out of distribution data [1, 2, 3] and (b) contextualizing progress on ImageNet (i.e., what does high accuracy on ImageNet really mean?) [4, 5, 6].
For (a), maybe surprisingly the answer is mostly yes! Better ImageNet accuracy generally corresponds to better out of distribution accuracy. For (b), it turns out that the ImageNet dataset is full of contradictions---many images have multiple ImageNet-relevant objects, and often are ambiguously or mis-labeled, etc---so it's hard to disentangle progress in identifying objects vs. models overfitting to the quirks of the benchmark.
[1] ObjectNet: https://objectnet.dev / associated paper
[2] ImageNet-v2: https://arxiv.org/abs/1902.10811
[3] An Unbiased Lookat Dataset Bias: https://people.csail.mit.edu/torralba/publications/datasets_... (pre-AlexNet!)
[4] From ImageNet to Image Classification: https://arxiv.org/abs/2005.11295
[5] Are we done with ImageNet? https://arxiv.org/abs/2006.07159
[6] Evaluating Machine Accuracy on ImageNet: http://proceedings.mlr.press/v119/shankar20c.html
Re: DeepMind achieves SOTA image recognition with 8.7x faster training
#30Earlier quoted context omitted.
Do you take advantage of previous iterations of the generator and discriminator? i.e. the generator should be able to fool all previous discriminators, and the discriminator should be able to recognise the work of all previous generators?
Nope! It's an interesting balance. The truth of the situation seems to be: the generator and discriminator provide a "signal" to each other, like two planets orbiting around each other. If you cut the signal from one, the other will rapidly veer off into infinity, i.e. collapse quickly. Or it will veer off in the other direction, i.e. all progress will stop and the model won't improve. So it's a constant "signal", yo…
Thanks for the detailed answer.
I don't claim it to be a novel idea, I just remember the Alpha Go (zero?) paper that said they played it against older versions to make sure it hadn't got into a bad state.