> until Hitler started incarcerating millions of people and threatened all out war on Europe, the US wasn't really bothered all that much
If by US you mean the public, many in the public did care, insofar they were aware. Generally not enough to want refugees in their country, but they still wanted to see the atrocities of the Nazis condemned by their government more than they were.
But if by US you mean the US government, parts of it were bothered enough to block reports, at the very least:
https://exhibitions.ushmm.org/americans-and-the-holocaust/ma...
> In early 1943, US State Department officials blocked reports about the mass murder of Jews from reaching the United States. Some at the State Department wanted to avoid any increase in public pressure to aid Jews and thought that if the American people did not have information about the atrocities, they would not protest.
[..]
> In November 1943, a group of congressmen, influenced by activist Peter Bergson, introduced bipartisan resolutions in the House of Representatives and Senate calling for President Roosevelt to create a government commission to rescue Europe’s Jews. In secret testimony before Congress, Assistant Secretary of State Breckinridge Long declared that the State Department already had admitted 580,000 refugees, a claim soon publicly proven false.
> At the same time, US Treasury Department staff investigated the State Department’s delays and obstruction in sending relief into Europe to aid Jewish refugees. They discovered that State Department officials had deliberately suppressed reports about the murder of Jews. Secretary of the Treasury Henry Morgenthau Jr. agreed to present their findings to President Roosevelt.
> As a result of Treasury’s efforts, Roosevelt signed an executive order on January 22, 1944, establishing the War Refugee Board. It was charged with the rescue and relief of victims of Nazi oppression as long as it did not interfere with the war effort.