Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Large Scale Visual Recognition Challenge 2011 - Results

vision.stanford.edu

Post reply on HN
Print view View original

21–30 of 38 posts

Re: Large Scale Visual Recognition Challenge 2011 - Results

#21

Post by [deleted] »

is74 wrote: post #17
pmelendez wrote: post #4

I don't think this proves a superiority of any algorithm against other. Just that SuperVision team did a great job on task 1 and task 2. I just would add two things: 1) There is a No Free Lunch Theorem ( http://en.wikipedia.org/wiki/No_free_lunch_theorem ) that had been applied to pattern recognition too and that states that there is not a significative difference in performance between most pattern recognition algor…

Many comments expressed concern about the alleged inappropriateness of the title. Even the no-free lunch theorem has been invoked, and words like SVM mentioned. However: The original title, "Neural Networks officially best at object recognition", is much more appropriate than the current title, because it is by far the hardest vision contest. It is nearly two orders of magnituder larger and harder than other contests…

[deleted]

Re: Large Scale Visual Recognition Challenge 2011 - Results

#22

Post by kylebrown »

sumodds wrote: post #6

Am not sure if you can apply winner takes all for such marginal difference in error. Give a slightly different database and things go awry. Check out : "Unbiased Look at Dataset Bias", A. Torralba, A. Efros,CVPR 2011.

Thanks for the reference. It goes well with "Machine Learning that Matters", a paper cited by Terran Lane in his recent blog post "On leaving Academia".

Re: Large Scale Visual Recognition Challenge 2011 - Results

#23

Post by [deleted] »

robrenaud wrote: post #16
pmelendez wrote: post #4

I don't think this proves a superiority of any algorithm against other. Just that SuperVision team did a great job on task 1 and task 2. I just would add two things: 1) There is a No Free Lunch Theorem ( http://en.wikipedia.org/wiki/No_free_lunch_theorem ) that had been applied to pattern recognition too and that states that there is not a significative difference in performance between most pattern recognition algor…

Isn't NFL utter crap? When you average an learning algorithms performance over a whole bunch of domains that _NATURE WILL NEVER GENERATE_, all algorithms are equally bad. Paying attention to the theorem is mostly defeatist and counter-productive. Imagine some ads serving company improves their learning algorithms 10% and is making 100s of millions more dollars. Are you going to say, well, there are billions of other…

[deleted]

Re: Large Scale Visual Recognition Challenge 2011 - Results

#24

Post by iandanforth »

Hinton's team (SuperVision) uses an interesting 'dropout' technique. He gave a Google Tech Talk on this back in June.

http://www.youtube.com/watch?v=DleXA5ADG78&feature=plcp

And an older talk that covers some of what a deep convolutional net is:

http://www.youtube.com/watch?v=VdIURAu1-aU

Re: Large Scale Visual Recognition Challenge 2011 - Results

#26

Post by kylebrown »

[deleted] wrote: post #23
robrenaud wrote: post #16

Earlier quoted context omitted.

Isn't NFL utter crap? When you average an learning algorithms performance over a whole bunch of domains that _NATURE WILL NEVER GENERATE_, all algorithms are equally bad. Paying attention to the theorem is mostly defeatist and counter-productive. Imagine some ads serving company improves their learning algorithms 10% and is making 100s of millions more dollars. Are you going to say, well, there are billions of other…

[deleted]

So you're asserting that the 10% improvement by Supervision is because they used the raw RGB pixels. Is that right?

If so, then I'm guessing that the other teams only used compressed representations like Fisher vectors with linear classifiers, because they needed to scale. Instead, Supervision achieved scale with raw power, doing the training computations on GPUs for a week (probably coded in openCL).

Re: Large Scale Visual Recognition Challenge 2011 - Results

#27

Post by jules »

sumodds wrote: post #6

Am not sure if you can apply winner takes all for such marginal difference in error. Give a slightly different database and things go awry. Check out : "Unbiased Look at Dataset Bias", A. Torralba, A. Efros,CVPR 2011.

The difference in error between the first and the rest is ENORMOUS.

Task 1:

    1st 0.15315 (convolutional neural net)
    2nd 0.26172
    3rd 0.26979
    4th 0.27058
    5th 0.29576
    [...]
Differences:

    0.10857
    0.00807
    0.00079
    0.02518
As you can see the first is way ahead of the rest. The difference between the 1st and 2nd is ~11%, between the second and third ~1%.

Task 2:

    1st 0.335463 (convolutional neural net)
    2nd 0.500342
    3rd 0.536474
Idem dito.

But the most exciting thing is that the results were obtained with a relatively general purpose learning algorithm. No extraction of SIFT features, no "hough circle transform to find eyes and noses".

The points of the paper you cite are important concerns, but this result is still very exciting.

Re: Large Scale Visual Recognition Challenge 2011 - Results

#28

Post by cf »

pmelendez wrote: post #11
pjin wrote: post #5

Earlier quoted context omitted.

To nitpick at the math: "No free lunch" results are asymptotic in the sense that they necessarily hold over the _entire_ domain of whatever problem you're trying to solve. Obviously, algorithms will and do perform differently over the relatively few inputs (compared to infinity...) that they actually encounter. It's similar to undecidability: just because a problem is generally undecidable doesn't mean you can't comp…

Agreed... I was in a rush to catch the train this morning and I didn't have chance to elaborate, I shouldn't do that. However, my point was that most of the algorithms used on that link (ANN, SVM, etc) had similar expressive power (VC dimension) and had been proved to have similar performance between them in object recognition. People normally take advantage on their specific properties rather than paying too much at…

That's why they include which features they used, which is educational.

Re: Large Scale Visual Recognition Challenge 2011 - Results

#29

Post by jules »

pmelendez wrote: post #14
is74 wrote: post #12

Earlier quoted context omitted.

Actually, it does, since the difference in performance between entry #1 and entry #2 is so huge (25% error vs 15% error!), and since this is by far the hardest computer vision challenge yet!

Sorry for disagree, but it seems more related to the fact that they are using deep convolutional learning rather than the neural network itself. If you use an ANN with the same set of features side by side with a SVM you will see very equivalent results. I will be more agree with a title like "Deep Convolutional learning overperformed traditional techniques in Object Recognition"

Yeah, if you use the same raw RGB features for the SVM as the neural net then the neural net would blow the SVMs away even more utterly.

Re: Large Scale Visual Recognition Challenge 2011 - Results

#30

Post by pmelendez »

kylebrown wrote: post #26
[deleted] wrote: post #23

Earlier quoted context omitted.

[deleted]

So you're asserting that the 10% improvement by Supervision is because they used the raw RGB pixels. Is that right? If so, then I'm guessing that the other teams only used compressed representations like Fisher vectors with linear classifiers, because they needed to scale. Instead, Supervision achieved scale with raw power, doing the training computations on GPUs for a week (probably coded in openCL).

"So you're asserting that the 10% improvement by Supervision is because they used the raw RGB pixels. Is that right?"

No, what I meant is that SuperVision did very well because their feature space is richer than the other teams, but IMHO that is resulting for the deep convolutional process which it used to generate rich features. This is a good explanation of the subject:

http://ai.stanford.edu/~ang/papers/icml12-HighLevelFeaturesU...

I deleted all the comments I could as they were complaining about the title. Since it was changed to a proper one most of my comments are not relevant any more.

Post reply on HN