The precedent you are looking for is about the legal principles that limit the extent to which the government can prevent free travel, not quarantine per se. Quarantine must conform to that precedent, being one of many examples where the government may restrict travel. The No Fly List is a modern example of a government policy that violates free travel and has consistently suffered under judicial scrutiny -- it is insufficient for the government to deem someone a threat to public safety to deprive them of fundamental rights. You can no more violate rights because someone "might be a terrorist" than you can violate their rights because someone "might have COVID-19" absent strong and specific evidence. There is much more latitude to limit what people do than where they go or what they say.
Americans have a strong Constitutional right to free movement within the country. As with free speech, suspending that right for an individual requires either due process or specific imminent threat to public safety to withstand judicial scrutiny. That is the calculus. A good test would be "can the government suspend free speech" in this scenario, as similar legal constraints will apply to free travel. That the specific case involves disease quarantine doesn't matter that much, as the Constitution makes no such exception. I don't follow freedom of travel case law closely, I am only aware that loads of it exists and that it is extremely biased in favor of the individual right.
As a related point, in the US most quarantine authority resides with the States, not the Federal government. If the States don't enforce it, there is little the Federal government can do.