Earlier quoted context omitted.
The problem though is that humans can adapt, and the AI in general can't. A go AI can play Go perfectly, but if a player overturns a board, it can do nothing to right this situation and resume playing. It's just following a list made for it, albeit a fairly substantial list. The worry is this, the cars will not fail like a human will. It may even be dangerous The car may not even be able to move in certain situations…
This. When these systems fail they will fail in surprising and inhuman ways. The accidents will appear to average people as easily avoidable which will give the impression of low quality engineering. e.g. Who doesn't slow down when a semi crosses their path on the highway?
It's an improvement as long as those are few percent of the unsurprising and human ways human drivers fail.