According to Apple maps, the doctor's office was literally in the middle of a field. I missed it the first time around (drove past it for a few miles, following my GPSs instructions) and then soon realized that I was already going through residential areas.
Never again, Apple. For context, this was 3 months ago. This doesn't seem like a hard problem to solve. My guess is that Apple just gave up.