Disclosure: my wife used to be on a work-permit scheme similar to H1B. The problem with all of these schemes is they're tied to a particular job. You give up that job, your right to remain is toast. This makes it hard to move to competitors and artificially depresses wages. In turn, this artificially affects native workers in the same industry. I think the solution here is more capitalism, not more government. Anyone…
They also make it very easy for skilled labour to immigrate. You just need a degree in your field and a sponsoring employer. There are no quotas, no need for the employer to prove they tried to find a local at above market rates, fees aren't onerous, etc.
The xenophobic view comes partly from how there aren't many skilled workers who want or can work in Japan (relatively low wages, Japanese language requirement) but mostly from their reluctance to allow unskilled labour in (which the US accepts via illegal immigration and Europe accepts via refugees)