I watched once a lecture on Coursera about Schelling's model of segregation, which I found extremely insightful. If you see maps like that with strong segregation, one of the first attempts to explain it is "racism". However, if you divide your neighborhood in 9 blocks such that you are in the center, you have 8 neighbors. Schelling's model now shows that even in cities where most people say:"I'd like to live next to…
Anyway, the equally interesting alternative to what you explained is what happens when everybody actually is super racist. If every person's requirement is that they refuse to live near anyone that is different, then you actually get a weird pattern where things are as mixed as they could be. The neighborhoods never stabilize, because there's always going to be border areas where each group can't help living next to a different group.
Actually, it just occurred to me that the model might not be accurate at that point, though, because people can't always move away like the model implies. Instead of people moving around chaotically, I wonder if the real result is violence intended to force the others to move.
EDIT: changed 'counterpoint' to 'alternative'