Machine Learning Confronts the Elephant in the Room

A system that can't recognise objects out of context is either completely useless, or only useful to the extent that it can recognise a very large number of objects in a very large number of varied contexts. Essentially, it's a system that learns primarily by memorising specific cases, rather than memorising and generalising to new cases.

This kind of system is limited by the number of objects and contexts with which it can be trained in practice. Which is why you need large datasets to train machine vision algorithms- because you basically have to show them many examples of everything you want them to learn.

In your example, the problem is not that the "AI" would not be able to recognise a car in space as a car, in space; it's that it would not be able to recognise it even as debris - unless it had already seen such "debris" before. Obviously, if it hadn't observed the behaviour of such debris it would also be incapable of predicting its behaviour, also.

The inability of statistical machine learning models to abstract and extrapolate from available observations, in the same way humans do, can be very surprising, especially in light of their othewise excellent abilities to recognise objects they have seen before.

For classification tasks, the labels are usually encoded as a one hot vector (one in the position of the correct class output, zeros everywhere else). If you don't normalize outputs to be between zero and one, it becomes a regression task - you are essentially asking the model to fit your one hot encoded label. That's not desirable, because we don't care about the actual value of the output for the correct class. Whether it is 0.1, 1.1 or 1001 it is the correct output as long as it's larger than outputs for other classes. That's why we want to take the largest output, and scale it in a way that it's always less than one. Its distance from one depends on how much larger it is than other outputs (the confidence of the model in this prediction).

Without normalization, the model that outputs 1000 for the correct class and tiny values for all other classes would get severely penalized because the labels says it should be 1 in that position (so the error is 1000-1=999), even though the model made the correct prediction.

There's some confusion about this (e.g. https://news.ycombinator.com/item?id=18054447 ), so hopefully my explanation makes sense.

https://ai.googleblog.com/2017/10/closing-simulation-to-real...

Simulating many years of robotic interaction is quite feasible with modern parallel computing, physics simulation, and rendering technology. Moreover, the resulting data comes with automatically-generated annotations, which is particularly important for tasks where success is hard to infer automatically. The challenge with simulated training is that even the best available simulators do not perfectly capture reality. Models trained purely on synthetic data fail to generalize to the real world, as there is a discrepancy between simulated and real environments, in terms of both visual and physical properties. In fact, the more we increase the fidelity of our simulations, the more effort we have to expend in order to build them, both in terms of implementing complex physical phenomena and in terms of creating the content (e.g., objects, backgrounds) to populate these simulations. This difficulty is compounded by the fact that powerful optimization methods based on deep learning are exceptionally proficient at exploiting simulator flaws: the more powerful the machine learning algorithm, the more likely it is to discover how to "cheat" the simulator to succeed in ways that are infeasible in the real world. The question then becomes: how can a robot utilize simulation to enable it to perform useful tasks in the real world?

The difficulty of transferring simulated experience into the real world is often called the "reality gap." The reality gap is a subtle but important discrepancy between reality and simulation that prevents simulated robotic experience from directly enabling effective real-world performance. Visual perception often constitutes the widest part of the reality gap: while simulated images continue to improve in fidelity, the peculiar and pathological regularities of synthetic pictures, and the wide, unpredictable diversity of real-world images, makes bridging the reality gap particularly difficult when the robot must use vision to perceive the world, as is the case for example in many manipulation tasks.

Note the bit about deep learning algorithms being very proficient "cheaters", which I missed in my comment above. Indeed, one way to fail to generalise from a simulation to the real world is to "overfit" to the defects in the simulation!

Like the linked blog post, most material you are likely to find online focus on training robots with (deep) Reinforcement Learning, I think because that just happens to be one domain where it is even harder to collect training data than good old supervised learning for image recognition. I can find virtually no source referring to the "reality gap" in the context of purely machine vision research- it's just not the done thing to train vision algorithms on simulated data, for the reasons described above, consequently it's very difficult to find hard data on why it's not done.

A good related source (with a bazillion references) is the following blog post, discussing the difficulties of deep RL, which is primarily trained on simulated environments:

https://www.alexirpan.com/2018/02/14/rl-hard.html

The post makes only passing referenct to the "reality gap" but it should give a good idea about the ins and outs of training in simulated environments.