Earlier quoted context omitted.
Voicing a different opinion is different than demanding someone be punished because they have a different opinion. Maintaining diversity of opinion is a two-way street that requires tolerance from everyone. Refusing to invite intolerant people is not hypocritical; it ensures the culture of spirited debate is preserved for all comers.
That is the exact same argument that the people you are opposing are making, just that you differ in opinion what is considered intolerant. And saying that you should try to exclude people who don't agree with you seems very much like "demanding someone be punished because they have a different opinion".
How would you design a system that is supposed to foster debate and differing opinions such that people don't shut it down because they cannot tolerate those differing opinions?