You're missing the thread here.
Suppose a guy walks into a grocery store with a shaved head and swastika tattoos. He wants to buy a bag of apples. The store owner doesn't like the cut of his jib and asks him to leave. That's fine, the guy goes to the grocery store across the street and buys the apples there.
The store owner across the street doesn't care what kind of tattoos a man has and is willing to sell apples to anybody. That's fine too, right? But then an angry mob descends on the store, filling the entrances and harassing the customers, accusing the store owner of being a complicit racist for doing business with Nazis. The store owner does some simple math and realizes that selling a bag of apples a month isn't worth this kind of trouble, so they start to turn the man away too.
Then the angry mob does the same thing to anyone who will do business with the man at all. He can't buy food, electricity, clothes, medicine, nothing, or the merchant will be the subject of a harassment campaign which will cost them more than they could get from his business.
This is clearly just as coercive as anything the government could do, if not more. Even they're not allowed to deprive you of food and medicine.
And as much as we might like the idea of making life impossible for Nazis, the target is only the current whim of the mob. Nothing guarantees that a given target is actually evil.
Or that they're paying enough attention to notice that the guy with the shaved head and swastika tattoos is a Buddhist monk and not a Nazi at all.
Which is why, when the government does it, they have an obligation to give you a fair trial and access to an attorney and respect your constitutional rights. None of which the mob gives you.