I can not, for the life of me, understand why that is such a sticking point with so many people. Fortunately I have enough people who are willing to work with me anyway that I can forget about the ones who don't, but it seems mind-numbingly obvious to me that if you're desperate for the skills I have, you should be willing to work with a requirement that doesn't cost you anything.
As near as I can fathom, it's a psychological thing. Remote people don't seem like they're part of your "empire" as you look out across the office and see all the busy beavers hunched over their computers.