A society where everyone is safe A society where everyone is free Choose one (Obviously real life is not so binary, but since freedoms are abused to inflict harm on others, removing the source of harm necessitates removing the freedom that led to it)
Similarly, my freedom to go out on the street and chant "gas the Jews! Gas the Jews!" would restrict the freedom of Jewish people to participate in normal life without fear. And I don't think these fears are ungrounded, or unreasonable, and can be simply dismissed.
Pretty much everyone is in favour of more freedom, across the ideological spectrum. The conflict is the disagreement where the balance lies between "give people the freedom to do what they want" vs. "people using their freedoms to restrict the freedoms of others".