Because it breaks your economy and social fabric (super simplified for the sake of fitting a decent reply into one short HN comment). We're at the lowest level of unemployment in decades, and wages are barely moving up. Bringing in large amounts of unskilled workers (remember, skilled workers get in on merit) would only delay wages rising (or those unskilled workers are going to be working under the table in jobs like slaughter houses, dairy farms, and other hazardous work environments where they will have no recourse when they're injured or maimed). Some immigrants want to integrate with American society, but not all (and that is what is expected of you as an immigrant, to share and hold our values in the same regard citizens do).
Europe has had its fair share of troubles with economic migrants. This isn't about race, this is about economic systems and the social stability of your host country. I support an immigration pipeline that gets skilled workers into the US as fast as possible (most especially in the healthcare field), but that's not what's on the public policy table.