Earlier quoted context omitted.
I agree entirely with your point, and wanted to follow a small tangent: > they lack self-preservation instinct I suspect that a self-preservation instinct, even in relatively dumb animals, is enormously complex. It might seem simple to us because it's so evolved, so hardwired. If I had to bet, I'd bet that a mouse's self-preservation instinct is more complex than the first-to-market Level 5 self-driving car will be.…
That's a good point, and yes, I definitely don't want an automated vehicle to have self-preservation instinct. What we're all after is people-preservation instinct, which is not entirely unrelated to the former. But the broader reason I brought up self-preservation is that in humans, it's not just about not getting killed in a crash. It's also about not having your life ruined by causing it, even if you walk away phy…
I think it might even be more like Platonic shadows on a firelit cave wall of the priorities of the company that made it.
At the current level that self-driving is at, the cars' control software is only able to perceive and respond to situations that its programmers anticipated and specifically coded for. That could be heuristics and rules in the hard-coded bits, but, also, even the most sophisticated deep neural net out there isn't even particularly capable of perceiving, let alone understanding, things that weren't coded into its training set. Meaning that, even if the company's intention was that the car should not plow into a pedestrian holding an enormous cellophane-wrapped Edible Arrangements™ fruit basket, that intention doesn't have any bearing on the vehicle's behavior if the software instead classified that object as a plastic bag blowing in the wind.