Here's my fear when I read your comment.
You're applying these labels to speakers (racist, transphobe, reactionary) and then claiming that if the label applies, engagement is pointless and they deserve to be shouted down.
Let's say I agree this is true, on net, for some set of beliefs. There's still some huge problems here:
1) How can we all agree which sets of ideas deserve this treatment?
2) How can we be certain these labels are being applied accurately?
Calls for violence are the only clear answer to the first question, but they're extremely rare - I'm guessing literally zero of the deplatformed speakers were actually calling for violence.
Beyond that, it's very hard to come up with a set of beliefs which we can all agree to. Socialism / communism killed tens of millions in the 20th century - should they be banned from speaking?
And who decides when these labels apply? I've personally seen examples of medical professionals using traditional scientific definitions of biological sex labeled as transphobes.
Given the difficulties drawing the lines, it seems better to have prudence when inviting speakers initially (fewer Milos) but then allow individuals to determine where those lines are themselves.