Speech is freer than ever before insofar as you have the opportunity to reach a vast global audience and insofar the trend in legal rights across the world as a whole has been towards having fewer restrictions on communication.
Being shamed isn't an infringement upon free speech. Being subject to legal sanction that impacts your liberty or property interest is. Your conservative friends have suffered no loss of their speech freedom, they simply haven't found a sympathetic ear for their viewpoint among liberals. Nor should they, say I as a liberal - when they come to me saying they want to restrict others' freedoms, I feel zero desire to assist them in that project or even associate with them.
Why would you assume that the subtextual nature of these fascist views explained their existence, though? They've been kept below the surface because when they were openly expressed in the past the overall reaction was highly negative.
You seem to think that if we just accepted people going about spouting fascist opinions, they'd eventually grow out of it or somesuch, as if it were a developmental stage everyone went through. The reality is that a lot of wars have been fought over these ideas already, most famously World War 2. And what we learned from that were that first, fascism when implemented on a large scale can lead to monstrous moral evils, as much as any other totalitarian ideology, and second, that some people seem totally OK with that.
Now, that would be merely interesting if we could be disinterested outside observers, in the same way that it's interesting to watch two ant colonies duke it out for territorial dominance because we don't empathize much with the individual ants. On the other hand, fascism (like all totalitarian systems) always has designated out groups who are designated to be acceptable targets for violence, both formal and informal. If you're passively accepting of that when it's happening to someone else next to you, then you share a small part in the moral responsibility insofar as your passivity differs from your base level of passivity. That is to say, if you're the sort of person who wouldn't lift a finger to help someone else in any trouble - being hit by a car, say, or suffering some wholly random natural calamity - then you're just an unhelpful person to begin with, whether that's due to anxiety, fear, or simple selfish indifference. On the other hand if you're normally helpful but don't help someone out if they're suffering from fascistic (or other ideological) violence, then you're indirectly assisting the oppression and bear some of the moral culpability for the results.
The fact is that when someone goes about saying (for example) they want the USA to be a country for white people and are willing to employ force to that end, that makes everyone who's not white feel much less safe because such speech normalizes the use of force against them. Of course, it's just as troubling if someone goes around asserting that they feel fine about hurting or killing white people. However, the facts that white people a) form a majority of the population and b) have inflicted far more violence against people of color in the recent historical past than they have been subjected to in return suggest a strong statistical basis for treating the first case as a much bigger problem than the second.