Thank you for contributing your perspective. I hear you. I don't have an easy answer; I can see the pendulum swinging the other way too, and I do worry about the precedent we might be setting. And I am thankful to live in a country that's (relatively) civil regarding LGBT freedoms.
But to return to the cake comparison: as an LGBT person as well, I still get screamed at on the street and harassed in public. Two nights ago I was walking with my same-sex partner and was followed down the street by a guy muttering, and then shouting, about "faggots" etc. He started catching up to us and we turned around, and he grabbed my partner by the neck, and we had to fight him off. But let's disregard the assault for a second and say he just followed us for a while, intimidating us, but technically within the confines of the law. Wouldn't you agree one side of this deserves to be stopped? To be silenced? I think it's an overly broad generalisation to say "Everyone always assumes it'll only work in their favour." I'm saying right here: I know that's not the case. But by the same token, trying to apply objective metrics to freedom of speech just as quickly justifies "I'm free to disagree with the way LGBT people live their lives" as much as "I'm free to walk in public without being afraid of assault." It's more nuanced than that. It isn't free or not-free. And I'd argue it's worth some people giving up certain freedoms (the freedom to denigrate other people based on their sexuality, for example) so others can increase theirs (like the freedom to walk in public without fear.)
So I'm glad I can get my cake, but if it means we're expected to shut up because we're afraid someone's going to roundhouse kick that cake out of our hands as soon as we walk out of the shop, that speech doesn't seem exceptionally free.