I'll bite, but I'd like to preface my response by saying that I'm not particularly a fan of Joe Rogan, even having watched a couple of his shows. The issue with saying that someone is a 'moderate liberal' is that that implies that there is a center to the extremes of the political spectrum somewhere; as the Overton window shifts (especially in the context of, say, hosting a very conservative guest) the relative political leanings of the people involved in the conversation will shift as well. (Thus if Joe Rogan is a moderate conservative who hosts a lot of extreme conservatives, he may come off of as a moderate liberal to his audience).
So with that out of the way, here's my response:
If you're familiar with the issues of Nash equilibriums and Mesa-alignment, you should know that a person's actions will sometimes contradict their inner motivations (e.g. peer pressure), or that people will compromise on their beliefs in the short run in the interest of preserving their beliefs in the long run (e.g. expressing moderate political beliefs when one's convictions run much deeper).
So in one sense, you're right: we can never know the 'true' inner motivations of somebody else. This is why we say "actions speak louder than words" (or thoughts); ultimately, the rubber hits the road we have to equate one's external actions with their internal beliefs.
I would say it's your right to express your opinion about what you think of others. I'm not arguing that 'because we don't know Joe Rogan's inner alignment, we can infer his political leanings at all.' What I am saying, though, is that Joe Rogan is not his podcast, and there are a number of reasons why a liberal talk show host would host more conservatives than liberals. I think it would be fair to say that Joe Rogan is generally in favor of free speech, and surrounding oneself with opinions that contradict one's own beliefs are a good way to refine how you express the beliefs you hold. Belief is ultimately determined by experience, but the same experience may result in different beliefs in different people (again, boils down to inner alignment).
While the people you surround yourself may affect your beliefs and political leanings in the long run, I believe that the guests on Joe Rogan's show vary wildly in their beliefs (e.g. he's hosted both Bernie Sanders and hyperconservatives) and Rogan's role in the podcast is mostly to act as a sounding board to help the other person express their opinions; therefore I feel that Joe Rogan's action of (supposedly) hosting more conservative guests than liberal ones does not imply that Joe Rogan is conservative.
But on top of that, the original spirit of my comment was to point out that the political lines we usually draw completely discard the nuance present in the real world. Some of the beliefs may be perceived as conservative, and others liberal. Projecting someone's complex (often hidden or inferred) political beliefs onto a relative political spectrum (e.g. liberal vs conservative) is a poor measure at best and purposefully tribalistic and reductive at worst.