Basically, I feel that completely removing the physical map is okay until you've picked a target. Then, having to click on it to be able to see what the route looks like (which streets to take, etc.) is higher friction than I'd like. Instead, imagine if hovering would give you a route overlay, and as you hover your mouse over multiple places you're considering, you're already aware of the physical directions as well.
Having to click back and forth feels quite constraining.
This is simply feedback on a way I think it could be improved further, not to take away from how good it already is.