As a Canadian, it's the snow that's made me skeptical of self-driving cars. Seeing how much they depend on road markings to navigate in traffic properly is completely impossible in winter. Until that's solved, there's a lot of places that just can't use them for a lot of the year.
Here's an interesting bit of research on filtering out snow noise from a LIDAR point cloud:
http://wavelab.uwaterloo.ca/?weblizar_portfolio=real-time-fi...