Remove various fundamental benefits such as public school, medical care, welfare, transit, subsidized X and the millions of other benefits created in the past 120 years and open borders are not as big of a deal anymore. You just need to hop on a boat and arrive at ellis island, like the ancestors of many americans today.
There are also parallels with city zoning laws. Go buy a house and find out how many restrictions and regulations there are on the parcel of land that you have purchased. And also see how much resistance you will get from the local politically active populace if you dare try to change things. Such as creating density. NIMBYism almost seems like a instinctual human reflex.
In other places where there isn't a welfare state and so on, it's usually straight up racism and ethnic nationalism.