I don't think you've understood my point. I'm not suggesting that selection doesn't occur, but rather that selection is only part of the story. Have a look at gene flow or genetic drift if you want to know more, they're both examples of non-selective processes that influence the genetic makeup of a population.