There's a strong counter argument that examines the history of the Nazi's initial rise to power and points out that they started very small, unfunded, unsupported and were able to gain power shockingly quickly. This is the basis behind Antifa's reasoning and tactics.
I'm not saying I agree with it, but when trying to feel out the contours of an issue like this, it's important to start at the edges.
The issue of when ideology and ideas become actively "harmful" or "dangerous" is an extremely complex one with a ton of grey area. It makes sense to start with the extreme cases where some of that grey area is removed. Those who argue that it makes sense to actively work to silence certain ideas argue it on the basis that those ideas represent a real and present danger to their lives and safety.
This certainly goes for Jewish people, like me, arguing against an ideology that promotes genocide of us based on our heritage -- I'm not even religious nor do I identify that strongly with jewish tradition, but based on the Nazi's accounting I'd have been sent to the concentration camps. People voicing support for those ideas are essentially saying I should be murdered for no good reason.
I hold it's reasonable to feel threatened by that and even to react to that with violence. Though it's not the strategy I adopt or recommend, it's one I feel a reasonable case can be made for.
Move in towards (ever so slightly) greyer cases. Take for instance people of color arguing against ideas that they feel would marginalize them for the color of their skin. What's the history, how much do those ideas represent a clear and present threat to these people's safety? In the case of those arguing in favor of, say, more militarized and aggressive police -- you could make a decent argument that this represents a real and present threat to black people's safety, given the prevalence of police shootings. Similarly, support of more profiling of brown people or requiring papers for immigration status. That represents a certain threat to the safety of perfectly legal brown people in the case that they are profiled and detained because they can't produce their papers.
All I'm saying that this is a huge grey area and that it is important to recognize that some ideas, beliefs, and ideologies are not harmless and do on their own represent varying levels of threat to other people.
Words are not harmless. The pen can be mightier than the sword and it can cut both ways.