>
What I never understand is what happens when a new situation that the system did not previously train for happen?I think most people ask this kind of question because they assume the entire system is some kind of big if-else-if-else statement. The question about swerving and hitting the kid or the grandma follows the same logic.
What we have to realize is that a neural net can and will react to things it hasn't specifically been told about using heuristics and "similar" stuff, just like human drivers. (see video of Tesla FSD avoiding a deer recently [1])
This is exactly like a new, or student driver. They have not yet encountered every possible situation, but they're able to apply reason and logic to new things and the more they drive, the better they get. So although they've never seen a deer jump out, they know it's good to avoid things when possible BUT NOT to swerve into oncoming traffic if there is traffic. So even a new-ish driver who has never seen a deer on the road will do a decent job of slamming on the brakes and swerving if save to do so. Same goes if a tire bounces towards you on the freeway, etc. etc.
Tesla are working towards billions and billions of miles of this kind of "learning driving", where the system keeps getting better and better.
And even after those billions of miles there will be situations it has never seen before, but the required response will be extraordinarily similar to things it has seen before. Chances are it will do a better job than you or I, because it's reaction time is close enough to 0 not to matter, and emotions won't cloud it's judgment if something really extraordinary happens (plane lands on highway, flood washes away road, etc)
[1] https://www.youtube.com/watch?v=pWJoe8hwu_I