The main problem is that immigration creates definite winners and losers. Even if you believe in the "right" of someone to move to a place that has more economic opportunity, what happens to the social dynamic and economy in the place they immigrate, and more importantly, from where they emigrate? Once upon a time people talked about "brain drain", but it seems we've either forgotten about it, or just don't care anymore, content to impose a second round of colonisation on the developing world.
The fact is, excessive immigration steals the brightest minds from developing countries and prevents their further development. It's great for the west (notwithstanding social effects); we get a never-ending source of labour that's willing to work for less than the indigenous population in the same field (salary arbitrage), while being able to draw from a pool of skilled workers far larger than our actual population.
So then this leads to the problem: it's great for the west, so our governments push migration. It prevents development of developing nations, keeping their living standards low. So what's the end game? I personally want to believe in a Utopian future where Accra could be every bit as developed as say, New York or Paris. Where migration isn't an economic issue. However if we continue as we have, migration will only cause more nations to become like Somalia, Libya, Syria; we steal all their human capital and, impoverished with no capacity to develop, they radicalise, we kill them with drones, and we're essentially living in a completely stratified world.
Anyhow, this worst case scenario probably won't happen because realistically, no single country can take in all the economic migrants of the world, but we do need to realise that when it comes to immigration, there are definite losers. Immigration is basically colonialism round 2. But it continues because, to the immigrants, it's an optimal personal decision to immigrate. For the host nation, they're accumulating human capital. The nation they're leaving loses.
And I'll finish with a quote from Thabo Mbeki, past president of South Africa, which I shamelessly ripped from a relevant Wikipedia page:
"In our world in which the generation of new knowledge and its application to change the human condition is the engine which moves human society further away from barbarism, do we not have need to recall Africa's hundreds of thousands of intellectuals back from their places of emigration in Western Europe and North America, to rejoin those who remain still within our shores!
I dream of the day when these, the African mathematicians and computer specialists in Washington and New York, the African physicists, engineers, doctors, business managers and economists, will return from London and Manchester and Paris and Brussels to add to the African pool of brain power, to enquire into and find solutions to Africa's problems and challenges, to open the African door to the world of knowledge, to elevate Africa's place within the universe of research the information of new knowledge, education and information."
And a page that touches on the issue of human capital flight: https://en.wikipedia.org/wiki/Human_capital_flight