Next step. Make it 3D using Street View. Theoretically, you have enough 3D information in there to detect building collisions. Also, Oculus ;).
There was a Google project that done this. I'd guess five years ago, so before full 3D buildings. Just hills and perhaps certain landmarks.