"It’s commonly assumed that freedom of speech is about the right of the speaker to express their ideas"
... without government interference. However, it is fully within the rights of the audience to condone, chastise, avoid, and laugh at the speaker.
To prohibit the audience from that reaction would be to suppress their free speech.
Freedom of association also requires that a group be able to prevent speech in the context of a group assembly.
"Who decides what to censor?"
Who is calling for government censorship of which expression? This essay sounds like a completely different discussion on free speech than what I am used to hearing, and I don't understand the context.
"If you ask, “Who should decide which ideas you may hear or read?”, people boggle."
Most people don't seem to boggle about porn filters or automatic spam rejection. The books "Ulysses", "Lady Chatterley's Lover", and "Memoirs of Hecate County" were once banned in the US.
While I don't agree with those bans, I bring it up to point out that people don't always boggle.
"So we ban Mein Kampf."
Who is proposing that we ban it? I understand that it's rhetorical, but it doesn't sound like a serious example of anything in the current discourse.
As I understand it, the question is, should organizations like a public university have some ability to override a student group which has selected a contentious external speaker to speak, given that the event uses university subsidized facilities and security staff.
That's hardly calling for a ban on Mein Kampf.