To avoid their society devolving into "rule by overt threat of violence", people typically agree that someone who says "I will shoot John Doe between the 3rd and 4th ribs with the FAL I have in my attic using this .308 round at 2:13 pm tomorrow as he walks his kids to school like he does every day except Tuesday, unless he pays me $500" should be able to be stopped, violently if necessary, before they have a chance to actualize their speech.
Arguing whether that speech is, in-of-itself, violence comes down to the messy semantics of trying to define violence. The fact that it is unacceptable is relatively unanimously agreed upon. So the question isn't "Can speech be unacceptable?" it's "What kind of speech is unacceptable?".
The example I gave was a specific, actionable, and credible threat against an individual, by the individual that intended to carry out the threatened violence. I'll assume most readers would mark that as "acceptably provoking punishment". Where it gets messy is when you start shaving off qualifiers. For example, someone stating "I'm going to kill all the programmers, and here's how" is potentially making a credible and actionable threat, but against a non-specific group of people. Is that worthy of punishment? What about someone with a large audience saying "It is our moral obligation to eliminate the programmers by any means necessary", maybe wink-winking and nudge-nudging at a guillotine in the background. It's not specific, and they're not saying THEY are going to be the ones to commit the violence, but it's still a little hinky.
In my opinion, that last one is probably the source of most discourse around "is speech violence". Most everyone is okay with suppressing more overt threats than that. It's when you get to that line that people break down into the camps of "people using their influence to benefit by threatening violence in a plausibly deniable way" warranting (self)defense or not.
So saying "but my free speech" or "it's just speech" isn't really a compelling point. We all agree that some kinds of violent speech should be suppressed. The question is simply where the line gets drawn.