>> Really all that matters is deaths per passenger mile, and weighted far less, injury/crash per passenger mile.
That's not exactly right. You need to take into account how likely it is for accidents to happen, not just the number of miles travelled. If the low probability of accidents is taken into account it turns out it takes many more millions or even billions of miles than already travelled for self-driving cars to be considered safe. See:
Driving to Safety
How Many Miles of Driving Would It Take to Demonstrate Autonomous Vehicle Reliability?
Given that current traffic fatalities and injuries are rare events compared with vehicle miles traveled, we show that fully autonomous vehicles would have to be driven hundreds of millions of miles and sometimes hundreds of billions of miles to demonstrate their safety in terms of fatalities and injuries. Under even aggressive testing assumptions, existing fleets would take tens and sometimes hundreds of years to drive these miles — an impossible proposition if the aim is to demonstrate performance prior to releasing them for consumer use. Our findings demonstrate that developers of this technology and third-party testers cannot simply drive their way to safety. Instead, they will need to develop innovative methods of demonstrating safety and reliability.
https://www.rand.org/pubs/research_reports/RR1478.html