I make use of the strawman-y American terms because it's the easiest way to avoid getting into long drawn-out arguments about what words mean[0]. So I interpret "open borders" to mean "any immigration policy that is permissive enough that an unqualified migrant with no criminal history can enter a country, apply for a job, and be hired without any additional bureaucratic formalities". Ellis Island, essentially.
"Left" and "Right" are also effectively unmoored from definition (unless you happened to be sitting on a particular side of the room in the Estates General of 1789), so I'm using the Political Compass's definition of those terms. It's fairly safe to say that there are enough commonalities in rhetoric between, say, Kropotkin (left-libertarian) and Lenin (left-authoritarian) that you can put them on one side of a two-dimensional plane and not get laughed out of the room. Likewise, on the right, Mussolini (right-authoritarian) and Rothbard (right-libertarian) didn't agree on much either, but they both wanted what power did exist to be invested in corporations explicitly empowered to put profit over people. You can also make a pan-authoritarian pairing between Stalin and Hitler[1] and a pan-libertarian pairing between Kropotkin and Rothbard, with about the same level of in-fighting on all four sides.
Yes, the Political Compass's definitions have Problems, but they are useful enough to make my point, which is that people whose ideology skews against state power also tend to oppose the state imposing restrictions on entering or exiting a country.
If you want the best argument against my point, it's that right-libertarians can't decide if immigration is a human right to be supported or a welfare program to be abolished. Their argument tends to be something like "Social Security or Ellis Island, pick one". This is a false dichotomy, of course[2], but that wasn't my point. My point was that the American people generally do not care about immigration beyond "we need to get these scary-sounding South American gangs off the streets". America has done a bang-up job of isolating their people from immigration bureaucracy in such a way that most people aren't even aware of how authoritarian the current system is. People talk about "moving to Canada" like it's something people can just unilaterally decide to do, and as if Canada's immigration system wasn't morally equivalent to a rock and a sticky note that said "YOUNG COLLEGE-EDUCATED CHINESE, FRENCH, AND BRAHMIN INDIANS ONLY, ALL OTHERS NEED NOT APPLY".
[0] https://xkcd.com/1984/
[1] Insert joke about the Molotov-Ribbentrop pact here.
[2] In practice, the way the EU handled this concern was to let member states put a waiting period on social benefits to prevent migrants from shopping around for the best welfare program to use.