From a public confidence perspective, it doesn't matter if a self-driving car crashes one tenth, even one one-hundredth as often as human drivers.
If you see a self-driving car cause an accident, particularly a lethal one, in a situation that almost any human driver would have avoided, you've totally destroyed any and all confidence in this car's driving ability, because "I would never, ever have crashed there."
As we've seen, there are lots of scenarios like this. The Tesla crash from last year, where the car simply didn't see a white truck against a light background. Or imagine an adversarial image attack, where some tiny insignificant detail is placed onto a stop sign or a "do-not-enter" that turns it into nothing from the perspective of the AI driver.
These kinds of scenarios obliterate public confidence in self-driving cars, because intuitively, you immediately realize that you're "a much better driver" than this car! Even if that's untrue 99/100 times, it only takes one visceral example to drive this kind of wedge.
Self-driving cars don't just have to be better than human drivers. They have to be as close to perfect as is possible, because that's what people will expect.