No, he’s right. It was very much in the interest of rulers to rule over as large a population as possible as that meant more tax income.
Think of all the slaves that were brought into the Roman Empire for an example. Or the laws binding serfs to the land in the Middle Ages. Or of the early modern kingdoms of South East Asia, which were essentially slave states, raiding their neighbors for their population so they could at them to their own.
That’s one aspect of it.
The other is that despite travel being difficult, historically people have done a lot of it. This has become clear in recent years as DNA analysis of ancient populations has become possible. The origins of most Western Europeans lie in the steppes of Ukraine (it might not be clear who built Stonehenge but it’s certain it wasn’t an ancestor of a modern Briton). There were Indian traders in Rome (probably to sell silk and spices) and Dutch settlers in South Africa were witness to Bantu immigrants sweeping in from he north.
People have always been mobile, it turns out.