I think this is just complacency and a bias towards how things are; imagine a world in which biking is safe, pedestrians are safe, and huge amounts of the city don't have to be devoted to parking. That's what 24/7 driverless cars can enable.
Why will self driving cars enable this though? It seems to me that an equally plausible outcome is that we move even further towards cities built solely around the car.
If I'm short on time, going to the airport, or my eyes are dilated from going to the optometrist or something, I'll use Lyft or Uber. This is like a once a month or less occurrence, (as you might guess since dilated eyes are one of my main reasons for using a car... I don't go to the optometrist that often.)
So I'm a world where few people own care, you might expect something like this. The car costs are up front per trip, not sunk as they are when you own the car, so you should see people thinking more rationally about their trip planning.