Honestly I think this just doesn't work out this way in reality. First of all, we know that deplatforming racists/fascists does actually help to stop the spread of racist/fascist ideas.
Secondly, and I think this is the more salient point, encountering racist/sexist/transphobic/whatever speech as a person who is the target of such rhetoric is immediately harmful and othering, especially when these views aren't actively challenged or disavowed by society. This doesn't even include the possibility that hearing hate speech might trigger some prior trauma experienced because of one's identity.
By taking this stance, you're implicitly advocating the allowance of direct threats to other people's very existence, under the justitification that everyone will be better off that way because it's all out in the open. To me, that just sounds like you don't really understand the impact that hate speech can have on a person's ability to live with the same freedoms and opportunities that other members of society enjoy.
Example: You have a job interview downtown, but you're black and the Klan has a parade scheduled that day, and you're scared of the calls for violence. You don't make the interview, so you obviously don't get the job. You were denied the opportunity to because of the 'free speech' of others.
I for one would like to see more focus on Freedom of Association alongside Freedom of Speech in discussions like these - you can argue whatever you like, but nobody has to listen to you or give you a platform (and if you're a fascist, they shouldn't, and should be actively deplatforming you because your ideas are so awful/dangerous).