21–30 of 43 posts
It seems like, to be really good the AI needs to construct an internal 3D model of objects so it won't matter which way it's rotated. It seems to be how the human works. I can rotate an object in my mind and picture it from any angle.
Being able to measure our successes and failures is the first step towards better algorithms :) This is always exciting. I don't think anyone thought that vanilla CNN architectures were all that would ever be needed. This paper which received an honorable mention this year from NeurIPS conference first attempts to convert the image to a 3d scene before detecting objects. https://arxiv.org/pdf/1906.01618.pdf
The thing about AI/CV and other interpretation simulators is that there is always a quantization of nature in the end result. This is why the Uncanny Valley exists, "Uncanny Valley" being a term that refers to the difference between nature and the model itself, and I'm pretty sure it can't be engineered out of the technology. That is, the simulator can not picture an object from any angle, only an angle greater than…
That is incorrect. From the Wikipedia page you references:
as the appearance of a robot is made more human, some observers' emotional response to the robot becomes increasingly positive and empathetic, until it reaches a point beyond which the response quickly becomes strong revulsion. However, as the robot's appearance continues to become less distinguishable from a human being, the emotional response becomes positive once again and approaches human-to-human empathy levels
> The thing about AI/CV and other interpretation simulators is that there is always a quantization of nature in the end result.
That's just not true in any meaningful sense. Computer vision can process images with a higher resolution than the human eye can distinguish.
> So, "construct an internal 3D model of [something in the natural world]" will always be deficient, and any conclusions derived from these models will always have inherent errors
Humans do this too (hence optical illusions). There's no reason to think that machine models can't surpass human models (and in some domains they already do).
All models are wrong, but some are useful.
I wonder if the exact some models that failed this test would succeed if their training data included images of weird angles/unusual contexts? My guess is every "hammer" image in the training data set was "conventional" -- a convenient angle and orientation. If half the images of "hammers" were instead "unconventional", would the model adapt to realize "my existing model of a hammer is incomplete; there must be a way…
Nevertheless, I agree with you. Given a huge dataset with millions of of unconventional images may be enough. Who knows.
Things kind of go in cycles in machine learning (similar to other fields). There is nowadays growing dissatisfaction of having to use so much (labeled) data, and people want the models to be better "primed" to capture the variations and structures existing in the real world. Partially because labeling a lot of data is just very expensive, but partially it's also seen as inelegant and "black-boxy" or it's just not in their scientific taste.
Other people argue that learning it all from data is fine and this kind of robustness shouldn't have to be baked in to models. Rather they should/could be learned from vast amounts of unlabeled data instead (Yann LeCun seems to be in this group.), with unsupervised/self-supervised methods.
What are the implications for ML used in real-world situations with no training set (or a very limited one) that could have life or death consequences - passenger vehicles, industrial use, military, etc.? Or is the consensus that it is a matter of time before compute and algorithms make these situations "safe enough," even for edge cases?
When a model is trained on ImageNet the training dataset is (usually) enlarged by doing artificial image augmentation. This does things like rotate, skew, crop and recolor the images so the model understands what the object can look like.
This dataset appears to find angles of objects that are difficult to reproduce using this process.
That is useful, but I can think of two ways to solve this pretty easily that would be achievable and would make a good project for a undergrad or Masters student.
1) Acquire 3D models of each of the object classes, render them at multiple angles in Unreal (or similar) and augment the ImageNet dataset with these images
2) Assuming you want to use the whole ObjectNet dataset as a test set, follow their dataset construction process using Mechanical Turk, and train on that data.
I bet either of these processes would take back 20-30% of the 45% performance drop very easily, and I bet the ones left would be the ones that humans have a lot of trouble identifying.
It seems like, to be really good the AI needs to construct an internal 3D model of objects so it won't matter which way it's rotated. It seems to be how the human works. I can rotate an object in my mind and picture it from any angle.
I think humans use both strategies. Sometimes we really rely on superficial visual information, like yellow/black stripes -> time to get away! No need to first perfectly match the visual input to a mental tiger model rotated at the correct orientation. I think split-second recognition is usually like this. Or perhaps we use different strategies for different objects, I could imagine for example that facial recognition in the brain is more 2D-feature based pattern matching, rather than 3D reconstruction.
Earlier quoted context omitted.
You can, but I don't think that's what happens when looking at stuff. When I look at the hammer, I don't rotate a hammer to see what orientation matches what I'm seeing, I see the handle and... metal bit (don't know the term, sorry), realize those look like they belong to a hammer and go "oh yeah, it's an upside down hammer".
It took me a while to recognize that wooden chair. At first I tought it was a wooden hammer on top of some squared background, but then I realized it was a chair seen from above. For me, I actually imagine these objects moving/rotating to make sense of them when seen from unusual angles. That hammer you described, I look at it and imagine myself flexing it.
We do heavily rely on context.
I bet if you gave a human a short amount of time to identify these images, they'd have some mistakes too. Particularly the middle top one on the article
... it's a chair. Wow that took a minute. Some of these are devilishly tricky; clearly they're designed to hit all the difficult edge cases within the domain. What a fun dataset!
This looks like a nice dataset, but as someone who works in an adjacent field (ML on text) it doesn't seem as revolutionary as it is being presented as. When a model is trained on ImageNet the training dataset is (usually) enlarged by doing artificial image augmentation. This does things like rotate, skew, crop and recolor the images so the model understands what the object can look like. This dataset appears to find…
Earlier quoted context omitted.
It's clearly delusional to think that computer vision will go from soft computing (erroneous) to hard computing in less than a decade, at the rate of current incremental improvements. We will soon hit an accuracy wall that only breakthrough research will allow to beat. The problem being: there's too much research exploring the search space in the same direction and not enough foundational research.
I disagree. There is a huge amount of research on adversarially robust classifiers and detectors going on. One can also programmatically test a neural network on real data, synthetically damaged data, fully synthetic data, and adversarial data, and everything in between. You can statistically ensure you get any desired accuracy level on those tests. While that's not a hard proof of anything, it can allow you to be ve…