The real problem with self-driving cars is the car to human handoff. Over the long term it's incredible unlikely that humans will be any good at all at remaining aware and 'up to speed' on the current situation in the event that the car needs to give control back to the driver due to road conditions, hardware failure, or sudden situation that it cannot contend with.
This is still facing regulatory hurdles, obviously.