There's also a lot of unknowns and serious challenges and difficult problems to solve before driver-less vehicles can become "mainstream".
Majority (all?) of the road testing has been on fairly controlled roads with "normal" circumstances, ie. not a blizzard with iced roads, hydroplaning, debris falling in front of the vehicle at speed, etc.
Longevity of the vehicle - it's doubtful a driverless vehicle purchased today will still be on the road (and safe!) 10-15 years from now (there's an awful lot of computing and sensors in place, but would a computer that old still perform safely -- ie. would you use your computer from 15 years ago as your daily-use work machine? no - so why trust your life to it?), whereas majority of vehicles on the road today are 10-15+ years old. Sensors are expensive and prone to natural wear, etc.
What about sensors frizzing out and providing false-readings? Similar circumstances have destroyed $100 million spacecraft missions, let alone some $20K vehicle that's several years old and neglected by the owner. Even military-grade electronics and sensor arrays are not "fool proof" and totally "failsafe" -- and you know the quality, robustness, redundantness, etc.. of the electronics going into self-driving vehicles are not on par.
The moment the first human casualty occurs do to some software/hardware failure, there will be serious ramifications for the self-driving world.