Human drivers can feel relatively safe around other cars because of mutually assured destruction. You are both afraid of dying if you make a significant mistake.
Self-driving cars will never have this fear holding them back from making mistakes. They don't care if they die. How can you feel safe around that? You could argue there is actually an incentive to let them crash once in awhile, because the valuable real world data will be phoned home and used to improve the product for the small price of a human life.
Once corruption sets in down the road, cars with autopilot capabilities will simplify kidnapping and assassination. Simply activate the backdoor remote piloting feature and crash your target's car (or the Uber car they're a passenger in) into a tree at 70 mph, or lock the doors and drive to a pickup location to complete the kidnapping.
Driving in a traditional car wouldn't protect the target, because a group with ubiquitous optional remote control over self-driving cars could also force surrounding cars to enter kamikaze mode and ram the target off the road.
Governments will demand some kind of backdoor like this at some point. It's not a question of if, but when. At that point, we'll have to trust that they'll never abuse it (lol) and that no third parties will ever be able to hack it remotely.
Even if you think the government is your friend, the first worm using a remote code execution exploit that hops from car to car via Bluetooth and activates its payload on a specific date will be able to kill tens of millions of people.