Yeah, nah. Even when you put a self-driving race car on a closed track with no other cars, a human driver still beats its time by a handsome margin. In challenging conditions (visibility/rain/ice/etc) the human will absolutely crush the computer.
I'm extremely confident that self-driving cars will never make a significant improvement on car safety.
I'm also confident that it will never be applicable to anything but highway driving in good weather conditions - we will never see a Level 5 car, those designations have about as much practical relevance to us as the Kardashev scale.
First of all, the safety statistics that we have with modern cars today is such that you need to run 10 million cars for several years before you have gathered enough statistics to say whether a new development has actually improved the situation. For instance if you take the Volvo XC90, there has not been a single fatal accident in the US or UK since the model was launched in 2002, with over 1 million such cars sold in those markets combined. That kind of lag in your dev cycle means it's extremely hard to do any sort of meaningful development of a machine learning system in a way that demonstrably improves safety.
Second of all, human factors such as being distracted by a cell phone or being intoxicated, plus not wearing a seat belt, accounts for the overwhelming majority of fatal accidents. It is, beyond a shadow of a doubt, much easier to build a system that detects such conditions and then deliberately slows down or stops the car. It's already being deployed today and will only increase in the future, so most of the potential safety improvement from self-driving will be eaten up by other tech.