As a computing specialist who makes a
very generous income from a large US employer, I
am an expert in a specific technology. I've also been in the position of trying to hire developers with similar skill sets. As a result, I might bring a slightly different perspective.
I am a US citizen. As you point out, I don't need to dress "business casual" and I can (and do) work remotely, and the pay is good.
But in some specialties, developers are scarce in the US. They're probably also scarce worldwide, but the greater wages available here could entice them to immigrate.
In my niche in particular it's hard to find good developers for anything like reasonable wages. And in my specialty, it's not possible for a typical "generalist" to just hop in and learn quickly; for what I do, you'd need a broad background in C, C++, cross-platform issues, Windows, Android NDK, and Unix-ish development environments, as well as real-time/game experience. Someone missing any of those would have a steep learning curve.
I hear stories of developers making $500k+ per year, or getting job offers worth millions. At that level, then yes, I think we could stand to see a bit of downward pressure on salaries. I'm not quite there myself, though at my salary's current rate of growth I could see hitting that threshold in ten years or less.
That said, I do actually agree that H1B workers are often illegally used to pay below-market salaries. [1] I don't know how to combat this other than through better monitoring and stronger enforcement of the law. (For example, if an employer is found to be paying less than they should, forcing them to retroactively pay the difference in wages+penalties to any affected employees would be a strong motivator to pay them enough.)
Currently, H1B workers need to be paid 100% of the "prevailing wage"; maybe a higher threshold like 110% (or pick a number that makes sense) would convince critics that these workers are really hard to find in the US? Because if you can't find someone locally for 110% of the "prevailing wage" then that's a compelling argument that they aren't here.
[1] "Employers affirm in the labor condition application that the wage offered to the applicant is at least as high as that paid by the employer for the same type of job, and the number equals or exceeds the prevailing wage for the job in the same geographical area" http://www.immihelp.com/visas/h1b/h1b-visa-requirements.html
[edit] added comment about "hire a generalist" not being good enough