The UK has obviously seen a lot of debate about immigration recently, around the Brexit referendum. I've often felt like non-EU immigration was the elephant in the room during all this. 'Leave' voters have often been portrayed as racist bigots by liberal 'Remain' voters. (I'm one of the latter, and I've been guilty of this myself at times). But if we extend the same logic to the whole world, isn't the EU basically a…
Without saying that this is right, the general logic is that the countries in the EU have met certain criteria (economic, legal, etc.) in order to join the club. The citizens of these countries have mostly similar standards of living, education, judicial systems, etc. Hence, allowing free movement is not a big deal because it is assumed the citizens of one country won't move en masse to another, and if a significant number of them do, they won't be a financial burden on the host country. There are also a bunch of reciprocal and cost-sharing agreements (e.g. medical services may be charged to the country of origin, etc.)
This whole framework and inter-country cooperation doesn't exist with respect to countries outside of the EU, and that's why these countries aren't willing to take a chance with universal freedom of movement. (Just a recent example comes to mind where a bunch of non-EU migrants were deported from Greece for various violations and their country of origin refused to accept them. In the EU any member state can deport/remove citizens of another member for certain violations and the country of origin is legally required to accept them.)