A big part of it is timezones and language barriers. Unfortunately in tech, the big hiring pools of engineers are:
1. USA
2. Eastern Europe
3. India
4. China
5. Western Europe (Much smaller than the above)
And in this situation, usually the main office is in the USA, and sometimes the main office is in Europe. And then you have to synchronize with teams across very separate timezones. 5 minutes becomes 1 day and it become very frustrating.
Then on top of that, you have to deal with the culture and language differences that can make communications very difficult. You quickly start noticing people will organize into native-language social cliques very quickly and that can be a hard problem to manage. You can get to the point that you say you will fire them if they don't start communicating with other people in the company other than people in their native language.
Eventually with these remote workers, because of all of this separation, you'll want to eventually bring them into the main office. Then you have entire other suite of problems to deal with, since immigration authorities usually require you to go through a visa song and dance bring these guys temporarily into the main office that is pretty close to just importing them permanently into the main office anyway.
So to deal with this entire problem, they say screw it and try to import them into one time zone / black hole called the main office. Immigrating into one country starts creating a melting pot effect where their language skills improve, they integrate with the local culture and language of the main office better and you don't have timezone differences to deal with.
So you say, why not say remote workers but they have to be in the USA only? And there are many that do say that! Unfortunately, latin america just doesn't make a lot of software engineers, and you get laws like 'hire 10 mexicans for every foreign worker in your company' that prevent you from establishing an office in the same timezone at a place with freeform immigration policies, because it probably doesn't exist.