The very first example given, of Rushdie having a fatwa against him, is framed as an example of trying to answer "Why should we defend his writing?" And the general conclusion is that we owe nothing to the "injured" party (in this case, Muslim clerics).
But the Muslim clerics issued their own violent speech. Their speech could have led to the assassination of a human being. Does that speech owe anything to the injured (read: dead) party? Should we allow speech to incite people to kill, "because intellectual liberalism?"
The answer, enshrined into law in most nations, is: no. We don't let you get away with killing people by incitement of violence. Your speech has consequences, and in many cases, you will be held accountable; if not by the law, then by vigilante justice. (This isn't an argument, it's a fact proved by the opposing argument)
You can choose to make the distinction that if you don't see physical harm, that that's not real harm, and thus, doesn't matter. But certain speech can in fact cause physical harm. The consequences of certain speech, by certain people, in certain contexts, can be debilitating, physically painful, nauseating, and drive people insane, even to suicide. In many cases it simply isn't a choice; your body just reacts to stimuli.
The simplest example is kids who kill themselves from incessant traumatic bullying. Even if you think those kids should "toughen up" and "just deal with it", that's not how the brain works. Some people in society are more vulnerable to suggestion and more likely to self-harm than others, and a non-stop barrage of hate and intimidation puts them in a traumatic state. The feeling of anguish is so much they commit suicide to stop the pain. Speech does do real harm, both directly and indirectly.
So, where does that leave us? Arguing over to what degree "verbal harm" counts as "real harm"? Waiting for someone to die, then deciding who, if anyone, gets the blame? This is a pointless morass, because the whole problem can be avoided by just not fucking with people intentionally. Just like with other crimes, sometimes people do harm by accident, and we deal with that in many ways. But when people do harm intentionally, that's a much more serious matter.
Thus we don't need this pointless philosophical diatribe to justify our ability to cause harm at will. If you're going to be a dick, then be a dick. But don't pretend you're doing it to defend some high-minded ideal. You're doing it because you're a dick, and there is no defense for that.