Well, the first argument is that it would not work. When the ten new EU states joined, only the UK and Ireland opened its borders and we were flooded with Polish people, about 500k which is roughly about 1% of the UK population. There was this report I read which stated that unlike the fears of UK citizens, these polish wave of migration has contributed to the economy more than they have taken in benefits, housing, nhs, schooling.
Now it has quietened a bit, I suppose who wanted to go went in that first wave.
Now, suppose we opened the borders to India, Brazil, Mexico, Russia, Pakistan, etc. The Mexican and Brazilians will probably go to the US rather than UK, the Indians and Russians will probably want to come to the UK rather than US. India is 1 billion however, Russia 300 million. If a fraction of these people came to the UK the country would be flooded and go into a period of complete transformation and utter confusion and most probably civil unrest as the primitive territorial instincts take hold. The Indians will then demand that their culture is respected and therefore push for laws rather than integrate and do what they do in Rome.
In principle it is a noble idea. That is why although the EU project has its downsides, most people think it unthinkable to leave the EU. We like being able to travel freely in this continent, to allow trade to be completely unrestricted, but we did not sit one day and say, hey let us open our borders to all the EU. It has instead taken about 60 years for this project to evolve to what it has and still not all of the geographical europe is within the EU. So perhaps we should gradually have an EU style project in a world scale. That is, I suppose, what the UN was meant to be, but it has effectively failed. Maybe now we should try and set up institutions and a judiciary for a United Western World, whereby people are able to move freely within Europe, US, Canada, Australia, New Zealand and then gradually incorporate Russia, India, Brazil, etc.