There will be unforeseen technical issues that are not apparent with a single test car or even small numbers of test cars. There will be accidents, and we'll have lawsuits against Google or other manufacturers. Those will have to play out, so we figure out who is really liable when a driverless car crashes. Trial lawyers will flock to these like sharks to a bleeding fish.
We may get to a point where driverless cars are safer than human-operated cars, and are accepted by the public. I think it's at least 50 years away.