1. Those who want to work from home and argue largely from that position. They say they'll be just as productive, you can write code from anywhere, having a more flexible schedule will make them happier and more productive and so on; and
2. Those that think there is more to your job than the lines of code you write. In even small companies (maybe even especially small companies?) culture is important. Culture transmits largely by physical proximity. Osmosis if you will. There is value in team camaraderie, whiteboard sessions, going to lunch with colleagues, sometimes just sitting around and shooting the breeze about whatever.
I fall very firmly into camp (2). This also applies to splitting teams geographically (common within Google) and, all other things being equal, you're better off having your organization in N locations versus N+1 locations.
Raises and promotions are more a function of relationships than anything else. Not being there decreases visibility and diminishes relationships. Or perhaps it's just that those who basically just want to write code see no value in and/or spend no time on building relationships?
So I think if you found a group of likeminded people that just wanted to put their heads down and write code then they could probably work together as an effective distributed team but as soon as you're in the minority in that situation you're losing out and (IMHO) it's not what's best for a colocated team anyway.