I'm really tired of re-explaining this (not necessarily on HN) so I ask you to forgive me for stating it very briefly. Also, I'm not going to defend this particular content below from disagreement because I'm tired - some other day I'd be happy to.
In a nutshell, Milo argues that queers should go back in the closet, women should get back in the kitchen, and colored people should go back where they came from. He consistently delegitimizes and intimidates people he dislikes (which seems to be most of them), thereby creating a 'chilling effect' that dissuades them from exercising their own rights - 'chilling effect' is a legal term of art that's worth looking up if you're not familiar.
Moving on to the affiliates, by which I mean those who identify with him or subscribe to his views, my argument is simpler: some of his supporters shoot people. Just over a week ago a protester against his appearance at the University of Washington was critically shot in the stomach. The shooter turned himself in to police, claiming that he thought he was being attacked by a white supremacist, but a review of his Facebook revealed him to be a right-winger who had messaged Milo to request an autographed hat an hour before the shooting. He's still walking free AFAIK. Likewise I'd point to the deaths of 6 people at the hands of a gunman in a mosque in Quebec city last week, and to the deaths of 9 people in Charleston last summer at the hands of a right-winger whose avowed intention (restated at trial) was to provoke a race war.
I could go on an on. Violent protest by anti-fascist groups and the like should always be questioned and examined, given the variety of motivations claimed - I don't lose much sleep when I see property damage take place, but that's cause it's not my property or business. It's not a form of discourse* that should be resorted to lightly or enthusiastically - we have sports for people who just need to blow off steam. However, where there is abundant evidence of violence being systematically deployed against persons - even if it's on a distributed ad-hoc basis rather than the product of a hierarchical command structure then I absolutely believe that it's appropriate to employ it in return in defense of self or others.
I reject the argument that some people make that people are never allowed to act violently until they've already suffered an injury (at which time it might well be too late to adequately defend oneself), Where a credible fear of violence exists, so does a right to defend against it. From a American legal standpoint, you might want to look into the concepts of 'fighting words' and the technical definitions of assault. If I brandish a hammer and say I'm gonna get you, happyrock!' you're generally not obliged to wait for me to actually hit you with the hammer before punching me in the nose.
Obviously I'm treating several complex subjects very superficially in a short internet comment, but I hope you can agree that these are things I've thought carefully about rather than just expressing a momentary emotional outburst.
* No doubt someone will say that violence is antithetical to discourse, but I agree with Clausewitz that 'war is the continuation of politics by other means.'