Freedom of speech is called
freedom of speech and not the “freedom to choose not to listen” because the freedom not to listen is implied by freedom of speech, for anyone that's thought about it. And it's easier to say
free speech than recite On Liberty. Shorthand is usually enough.
Deplatforming is censorship because, if done actively and exceptionally (i.e. discriminatorially and out of the ordinary routine) then it is the attempt to prevent the speaker from speaking and anyone wishing to listeners being able to listen.
It seems to me that you're conflating distribution with purchase in your analogy, which might explain your subsequent reasoning, though I find them to be quite different things. If you are going to conflate them then distributors refusing to distribute the goods of gays or blacks based on them being gay or black, given that they usually distribute goods from non-gays and non-blacks, that is not prejudicial discrimination?
I would not agree with that in a marketplace, and that wouldn't fit with my conception of freedom of speech either, given the analogy.
If, however, we were talking about private individuals and not distributors, then I agree that they should be able to choose who they purchase from or even sell to - in a private capacity, not as a business - whether based on prejudice or not. Just as with speaking and listening.