Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Object-recognition dataset stumped the world’s best computer vision models

news.mit.edu

Post reply on HN
Print view View original

31–40 of 43 posts

Re: Object-recognition dataset stumped the world’s best computer vision models

#31

Post by Datenstrom »

I have been working on this problem for the last year. As it turns out this problem is especially prominent when doing object detection in the real world on a drone platform. Besides a large number of angles/contexts being able to move in 3D adds a large number of scales also, and there is hardly any open training data besides the VisDrone dataset[0] which doesn't even address the scale issue.

It is certainly an interesting problem though. I can't talk much about my work but if anyone wants to collaborate on something open source addressing the core problem check my profile.

[0]: http://www.aiskyeye.com/upfile/Vision_Meets_Drones_A_Challen...

Re: Object-recognition dataset stumped the world’s best computer vision models

#32

Post by nl »

bonoboTP wrote: post #29
nl wrote: post #25

This looks like a nice dataset, but as someone who works in an adjacent field (ML on text) it doesn't seem as revolutionary as it is being presented as. When a model is trained on ImageNet the training dataset is (usually) enlarged by doing artificial image augmentation. This does things like rotate, skew, crop and recolor the images so the model understands what the object can look like. This dataset appears to find…

1) Training with synthetic data is definitely a thing in computer vision, exactly the way you describe. You can even throw a GAN on top of the results to make the renderings look less artificial. 2) They do something like this, in section 4.3 by splitting ObjectNet in half. They fine-tune (the last layer of) ResNet-152 (I wonder what happens if you fine-tune more layers) on half and test on the other half. This pushe…

> I wonder what happens if you fine-tune more layers

Generally it improves some, but most of the gains are in the final layer retraining.

But it's a lot more data hungry.

> They do something like this, in section 4.3 by splitting ObjectNet in half.... here's still a gap, but it can be plausibly argued that the gap would close up if we scaled things up by one or two orders of magnitude

This is interesting. It's worth noting that this training is on only 64 images per class, and it is unclear if they augment this in anyway.

Before retraining, the paper itself notes:

Classes such as plunger, safety pin and drill have 60-80% accuracy, while French press, pitcher, and plate have accuracies under 5%

It is worth noting that the plunger, safety pin and drill classes are ones that have multiple orientations already in ImageNet, while French press, pitcher, and plate are almost all the "right" way up.

To me this indicates this is simply a data problem - the model has never seen what an upside-down French press looks like so it gets it wrong.

Re: Object-recognition dataset stumped the world’s best computer vision models

#33

Post by mistrial9 »

nl wrote: post #22
rhizome wrote: post #20

Earlier quoted context omitted.

The thing about AI/CV and other interpretation simulators is that there is always a quantization of nature in the end result. This is why the Uncanny Valley exists, "Uncanny Valley" being a term that refers to the difference between nature and the model itself, and I'm pretty sure it can't be engineered out of the technology. That is, the simulator can not picture an object from any angle, only an angle greater than…

> This is why the Uncanny Valley exists, "Uncanny Valley" being a term that refers to the difference between nature and the model itself That is incorrect. From the Wikipedia page you references: as the appearance of a robot is made more human, some observers' emotional response to the robot becomes increasingly positive and empathetic, until it reaches a point beyond which the response quickly becomes strong revulsi…

humans do not have only one state, one performance criteria or one mood.. on the other hand, of course sensors can read data that humans cannot, for some set of hardware and inputs. I like the comment above. It is too early for large generalization.

Re: Object-recognition dataset stumped the world’s best computer vision models

#34

Post by cs702 »

Datenstrom wrote: post #31

I have been working on this problem for the last year. As it turns out this problem is especially prominent when doing object detection in the real world on a drone platform. Besides a large number of angles/contexts being able to move in 3D adds a large number of scales also, and there is hardly any open training data besides the VisDrone dataset[0] which doesn't even address the scale issue. It is certainly an inte…

I recently saw a paper (which I reposted here on HN a while back) that perhaps could help with your problem. The paper proposes an approach that sort of induces models to learn representations of objects that are good at predicting "the most agreed-upon" rotations of the inputs. I'm not explaining it well. Anyway, a model in the paper achieved SOTA on a change-of-viewpoint dataset with a really tiny number of parameters. Might be worth a look: https://arxiv.org/abs/1911.00792

Re: Object-recognition dataset stumped the world’s best computer vision models

#35

Post by AareyBaba »

jobseeker990 wrote: post #8

It seems like, to be really good the AI needs to construct an internal 3D model of objects so it won't matter which way it's rotated. It seems to be how the human works. I can rotate an object in my mind and picture it from any angle.

"I can rotate an object in my mind and picture it from any angle"

Except there was a wonderful thread about Aphantasia a while ago https://news.ycombinator.com/item?id=20267445

Several HN readers chimed in to say they have this feature. It would be interesting to know if this dataset would stump them.

Re: Object-recognition dataset stumped the world’s best computer vision models

#36

Post by pauljurczak »

tcgv wrote: post #12
StavrosK wrote: post #10

Earlier quoted context omitted.

You can, but I don't think that's what happens when looking at stuff. When I look at the hammer, I don't rotate a hammer to see what orientation matches what I'm seeing, I see the handle and... metal bit (don't know the term, sorry), realize those look like they belong to a hammer and go "oh yeah, it's an upside down hammer".

It took me a while to recognize that wooden chair. At first I tought it was a wooden hammer on top of some squared background, but then I realized it was a chair seen from above. For me, I actually imagine these objects moving/rotating to make sense of them when seen from unusual angles. That hammer you described, I look at it and imagine myself flexing it.

Same here. I needed almost 10 seconds to recognize the chair.

Re: Object-recognition dataset stumped the world’s best computer vision models

#39

Post by JMTQp8lwXL »

jobseeker990 wrote: post #8

It seems like, to be really good the AI needs to construct an internal 3D model of objects so it won't matter which way it's rotated. It seems to be how the human works. I can rotate an object in my mind and picture it from any angle.

The model hasn't ever picked up or held a hammer. How is it ever supposed to recognize a hammer in an unfamiliar context (e.g., when given a new image that is not representative of what it was trained against?). You can't train intuition.

Re: Object-recognition dataset stumped the world’s best computer vision models

#40

Post by Datenstrom »

cs702 wrote: post #34
Datenstrom wrote: post #31

I have been working on this problem for the last year. As it turns out this problem is especially prominent when doing object detection in the real world on a drone platform. Besides a large number of angles/contexts being able to move in 3D adds a large number of scales also, and there is hardly any open training data besides the VisDrone dataset[0] which doesn't even address the scale issue. It is certainly an inte…

I recently saw a paper (which I reposted here on HN a while back) that perhaps could help with your problem. The paper proposes an approach that sort of induces models to learn representations of objects that are good at predicting "the most agreed-upon" rotations of the inputs. I'm not explaining it well. Anyway, a model in the paper achieved SOTA on a change-of-viewpoint dataset with a really tiny number of paramet…

I have been looking for an excuse to use capsule networks. I can't investigate it as a possible solution at work because even if good results are achieved the performance problems[0] would prevent real world use. Definitely might look into that as a side project though. I have also been looking for an excuse to do something beyond like SENet in Julia.

Maybe a new benchmark like this is what we need to get out of the rut.

[0]: delivery.acm.org/10.1145/3330000/3321441/p177-Barham.pdf

Post reply on HN