I'm torn because on one hand, having been intimately acquainted with a trans person who suffered greatly, and having heard from them what their internal voice told them in duress (e.g., that they were "actually delusional", "trying to trick others", "convincing no one", and other harmful but thoroughly implanted negative self-talk), I would absolutely start a fight with someone for certain kinds of statements, especially in the presence of someone I expected had negative self-talk similar to what was being said about them. From that angle I agree that speech can be violent. I can also see it being exhausting, because I doubt there is an end to individuals willing to say such things. I could see that feeling like a Sisyphean trial.
At the same time, I think most speech that comes across this way is the product of ignorance or unmet emotional needs, and if you listen to people that say hateful things they will often weaken their view as soon as they realize you plan to actually listen to them and understand how they feel. Because of this, I recognize my instinct to start a fight as a result of how I feel, and not necessarily a productive course of action, and often results in worse outcomes (people doubling down, taking an angry response as justification of their action, etc). I also find that this builds trust, which is necessary for good faith conversation.
My real problem with 'speech as violence' comes down to the difference between moralizing and compassion; I believe that compassion will produce better outcomes. Further if it is about morality you have to decide the correct morality, and it is impossible to get actual agreement on that. If you did you would have someone disagreeing to test if you would allow them to think differently than you without attacking them.
There are indeed people that say vile shit to hurt others intentionally, for various reasons from demotivating someone competing with them, to sadism, to bullying (which I find is often someone who believes 'bad people should be punished' and that someone has done something 'socially bad' so that it is 'up to them to correct it'). For example, When a sibling of mine wore something that might be associated with being gay, he received harsh criticism from another sibling I knew cared about him. The criticizing sibling had been bullied, and believed his statements would dissuade what he saw as a social misstep. It was a failure of compassion. The form of the criticism was also exactly what the criticized sibling had thought about himself.
I've also found that sometimes when my first thought was 'wow this person is a terrible person', it turned out that they thought that other people thought there were a terrible person, and so they stopped trying to do things that might prevent that effect, causing more people to think it, causing them to care less, or possibly even say provocative things with resent. If you listen to such a person and sympathize with them - however hard it may be, even if you don't agree with them, it can completely break the script they are expecting to hear, and lacking a prepared response, you may get an actually vulnerable one.
Really, I think as soon as morals come up, and rightness or wrongness, empathy has failed. And it is hard because empathizing with people that have different backgrounds and views is really hard. People will talk about empathy all day long, but I rarely see anyone really willing to empathize with someone who is narcissistic or abusive, someone who lacks all social skills as an adult and is smug that they are well paid in spite of it, or someone who believes radially different on any topic to really care about. Can any of us sympathize really with someone who thinks learning programming is a waste of time and we should all learn 'real skills' like how to read a balance sheet or run a meeting? Or someone that believes SOAP is the greatest thing that has even happened and people complaining about it just don't get it? That JavaScript is the most consistently designed language that has even existed and should have monuments built in its honor? That open-source software is a communist fad that harms markets and shouldn't be legal?
For me, when someone plays music from their phone speakers on public transport it is very difficult for me to sympathize. When someone cuts my spouse off in traffic, they immediate cuss up a storm. When a relative makes a political statement that is a bit tasteless and you know they just skimmed a news article and have no idea what they are talking about, it can be difficult.
But I think people get it backwards when they think it is difficult and it is all these peoples fault. Fault is a moral determination. Really, it is imaginary. It is a fiction that helps society run smoothly.
I will share, to aid my point, an error I have made. The first is that when I was in HS, I littered soda cans, mostly to emphasize that I didn't care about rules or anything (what a silly time that was). Then one day I saw someone picking up trash, including something I had dropped that had blown aside. When I saw that, I felt really hurt, guilty and sad. To know that I had done that. To this day, I pick up every piece of trash I come across and carry it until I see a trash can. It was a bad thing to do, but if the person picking up the trash had come up to me and yelled at me, telling me I was a lazy piece of shit, that I didn't care about anyone else, etc. I think I would have had a very different experience, and a very different response.
I sincerely believe, and hope, that many of these individuals who make these comments have the potential to be the ones that later go to the greatest efforts to repair damage they see themselves as causing. But for this to happen, I think they need to feel it as guilt (internally and based on empathy), instead of as shame (externally, and based on moral error). And for that to happen they need to feel heard, and they need to be engaged with emotionally.