Counterpoint - nobody actually cares about traffic fatalities. Nearly 40,000 deaths a year in the US, and the majority of people get in their cars every day without ever thinking about this risk and go about their lives (or to put that another way, the risks are already so low as to be negligible to most people, and anything else within the ballpark of negligible is still negligible). Normalcy bias is incredibly strong and as soon as self-driving cars are "normal" people will get on board without thinking twice. Tesla is slowly acclimating people to self-driving, basically everyone is familiar with the idea at this point, and as soon as it's available and someone tells you it's "just as safe as driving yourself", most people will just go with it. Especially given how big the upside is - you don't have to deal with the stress of driving anymore, you can just relax in your car. Or in terms of getting a ride, maybe it's 1/4 the price of a taxi driven by a human. Sounds good, people will roll with it.
Of course the more it starts taking off, there will always be a vocal subset of the population that is strongly opposed to it, just like there are vocal anti-vaxxer groups and there were anti-seatbelt protests back in the 80's. But I can't imagine the naysayers having a very big impact on the progression of the technology, the upsides are just too enormous.