I watched once a lecture on Coursera about Schelling's model of segregation, which I found extremely insightful. If you see maps like that with strong segregation, one of the first attempts to explain it is "racism". However, if you divide your neighborhood in 9 blocks such that you are in the center, you have 8 neighbors. Schelling's model now shows that even in cities where most people say:"I'd like to live next to…
As a map of distribution, you would expect significant clustering from randomness alone. We've all seen the "which of these dot patterns is actually the random one?" tests, where few people guess the correct answer: the one with all the clusters.
If you take that clustered map as a seed, then add any affinities, you get positive feedback, increasing the clustering. Those affinities don't have to be racial selection but can just be in the general form of, "I make decisions based on what I hear, and I hear more from people I know than from people I don't know."
So, you hear from your cousin that there is a new apartment building opening across the street from him. You hear from several members of your church that X is the best school district. A factory in your town closes and the laid off employees discuss their options and decide to go check out a company in the next town that is hiring (with lots of reasonably-priced apartments nearby).
Just getting more information from people you know than people you don't know could be sufficient for Schelling's model, but add to that the tendency to get more information from people from your background (whether you actually know them or not), because it's likely to be more relevant to you, and you magnify the effect significantly. An urban liberal and his soccer-loving Mexican gardener don't have to dislike each other for the former to read more NY Times and the latter to watch more Univision, and the information about where the opportunities are as presented in those two media outlets are not the same.
This doesn't mean that there is no race-based component in deciding where to live. It just means that, as the Schelling model demonstrates, you can't tell how much there is by looking at how much clustering there is on a map.