Then how do you decide what is (or should be) a right and what is not? Suppose someone claims a right to gay marriage, or a right to employ only people of their race, or a right to bear arms - how does society (the citizenry, the legislature, and/or the courts) decide whether that is indeed a right?
Absent a holy constitution given to us by the divine, the way we decide what is and isn't a right is we look at its effect. Places that recognize a right to bear arms, for instance, do so because they believe that arming the citizenry protects them against abuses of the government. Places that don't do so because they believe that disarming the citizenry protects them against crimes from other citizens. To the extent that one of these views is correct and the other is not, it is not because they guessed wrong about the nature of the universe - it is because one argument is correct and another is not.
Free speech is not axiomatic. Free speech is a right (and I agree it is a right!) because it has particular positive effects on society, through the benefits of open and unfiltered public discourse.
And the same reasoning helps us define exactly what "free speech" is. We make significant restrictions on free speech - classified information, copyright and trademark law, slander and tortious interference, electioneering laws, unauthorized practice of medicine or law, fraud, etc. - in the expectation that those restrictions serve to benefit society, and in the understanding that if we were to allow these forms of speech, they wouldn't really serve the goals which we see free speech, overall, as helping. If I were to say "I am a licensed doctor and I think you should take two pounds of Vitamin D a day," we understand that the benefit of me adding that statement to the public discourse is nil, and the harm is great, and so we don't recognize that as protected by free speech.