The privileges and immunities clause says that states must give the same privileges to citizens of other states that they do to their own citizens. Since states have always allowed free entry and exit to their own citizens (and indeed, the right to enter is intrinsic to the concept of citizenship), it follows that states cannot restrict the freedom of citizens of other states to enter the state. And the right to travel is moreover consistent with what the framers were trying to do with the Constitution: tear down internal barriers between states.
Privacy does not fit in there because no state has ever granted its own citizens the notion of privacy espoused by certain modern proponents of the "right." Nor can you read the contemporaneous works of the framers and see any discussion of this idea that people should be able to keep certain information private from the government. Instead, in the 4th amendment you see a much narrower concept: that the government shouldn't be able to trespass on your property without a warrant.
Not all rights are enumerated in the Constitution. But that doesn't mean you can make up a right and say the Constitution protects it. The right has to come from somewhere. Otherwise, "rights" become an end-run around democracy. Oh, ACA is unconstitutional because people have a "right" not to be forced to buy health insurance. Oh, you can't require background checks for gun purchases, despite the vast majority of Americans supporting them, because the Second Amendment prohibits that.